DATA INTEGRITY AND PLANNING FOR BUSINESS CONTINUITY April 2018

DATA INTEGRITY AND PLANNING FOR BUSINESS CONTINUITY April 2018 Stuart Jones – Director, Manufacturing Lead, Data Integrity Centre of Excellence

Information in this presentation draws upon a variety of sources, including published regulatory and industry presentations, policies, and other regulatory guidance documents, personal and client experiences, documents, and research, all or any of which may or may not have been prepared or conducted by the presenter. The example scenarios reviewed provide possible mechanisms to mitigate continuity risks; the recommendations included are for organizations to think about working towards and are not intended to be definitive, comprehensive, or appropriate for every organization

Agenda 1. Business Continuity Planning – Why? – Fourth level 1. Regulations Click to edit Master textIndustry styles 2. and Standards Fifth level a. Second level 3. Business Continuity Program and Business Continuity Plan Third level 4. Differentiation of Business Continuity Plan and Disaster Recovery Plan 5. Business Continuity Planning 6. Business Continuity Execution Process 7. Case studies 8. Summary 9. Q & A 3

Business Continuity Planning - Why? Must be able to continue running the critical processes during a crisis and restore operations 4

Business Continuity Encompasses planning and preparation to ensure that an organization can continue to operate in case of serious incidents or disasters and is able to recover to an operational state within a reasonably short period. Resilience Recovery Contingency Critical business functions and the supporting infrastructure must be designed in such a way that they are materially unaffected by relevant disruptions, For example through the use of redundancy and spare capacity Arrangements have to be made to recover or restore critical business functions that failed The organization establishes a generalized capability and readiness to cope effectively with whatever major incidents and disasters occur, including those that were not, and perhaps could not have been, foreseen. Contingency preparations constitute a response if resilience and recovery arrangements should prove inadequate in practice. For example alternative Business Process or Suppliers 5

Applicable for Critical Operations and Systems Supply Chain Physical Premises Information Technology Marketing Human Resources Disruption to internal sites Disruption to partnered sites Disruption to Logistics Disruptions at key suppliers Natural disasters such as fires, earthquakes and floods Terrorist Attack System Corruption Severe Crashes Cyber Attack/Hacking Virus Fraudulent activity resulting Reputation loss Recall Strikes Critical resource loss or reduction Reorganization 6

Critical Pharmaceutical Systems That Require Business Continuity For each Process describe the impact on the organization if the process is incapacitated. System that support life-saving processes or which execute a time-critical regulatory process Product Release QC Testing Lot recall Pharmacovigilance Also consider reliance on supporting information and interfaces with other processes : Change control Quality control Practical and procedure management , 7

Regulations and Industry Standards Supporting Business Continuity Plans ( BCP) and Disaster Recovery Plans ( DRP) Implied Implied Eudralex - Volume 4 - Good Manufacturing Practice (GMP) guidelines- Chapter 1 Pharmaceutical Quality System 1.4 A Pharmaceutical Quality System appropriate for the manufacture of medicinal products should ensure that: (i) Product realisation is achieved by designing, planning, implementing, maintaining and continuously improving a system that allows the consistent delivery of products with appropriate quality attributes; (vii) Processes are in place to assure the management of outsourced activities. (xv) Medicinal products are not sold or supplied before a Qualified Person has certified that each production batch has been produced and controlled in accordance with the requirements of the Marketing Authorisation and any other regulations relevant to the production, control and release of medicinal products ; Implied Explicit Explicit Eudralex - Volume 4 - Good Manufacturing Practice (GMP) guidelines- Annex 11 Computerized Systems 7.2 Regular back-ups of all relevant data should be done. Integrity and accuracy of backup data and the ability to restore the data should be checked during validation and monitored periodically 16. Business Continuity For the availability of computerised systems supporting critical processes, provisions should be made to ensure continuity of support for those processes in the event of a system breakdown (e.g. a manual or alternative system). The time required to bring the alternative arrangements into use should be based on risk and appropriate for a particular system and the business process it supports. These arrangements should be adequately documented and tested 21 CFR 211 Current Good Manufacturing Practice For Finished Pharmaceuticals Sec. 211.68 Automatic, mechanical, and electronic equipment (b) Appropriate controls shall be exercised over computer or related systems to assure that changes in master production and control records or other records are instituted only by authorized personnel. A backup file of data entered into the computer or related system shall be maintained except where certain data, such as calculations performed in connection with laboratory analysis, are eliminated by computerization or other automated processes. . Sec. 211.180 General requirements. (c) All records required under this part, or copies of such records, shall be readily available for authorized inspection during the retention period at the establishment where the activities described in such records occurred 8

Business Continuity Program and Business Continuity Plan Business Continuity Analysis Maintenance A Business Continuity Program is the organizational approach to develop, test, and maintain a Business Continuity Plan. Without an ongoing Business Continuity Program , the Business Continuity Plan is only a point in time and will be come outdated at a time of great need. The Business Continuity Program should integrate the Business Continuity Plan with ongoing GMP validation and Quality Assurances activities. Solution Design Test and verify Implementatio n For GMP processes Quality approves sops for the alternative process. Backup restore and alternative computerized systems are validated Business Continuity Planning never ends. Periodic review of back up and restore to ensure resiliency. When normal operations processes and procedures are revised or modified, the business continuity plans must also be assessed and revised. 9

Differentiation of Business Continuity Plan and Disaster Recovery Plan Business Continuity Plan Disaster Recovery Plan Process of defining arrangements and procedures to enable an organization to continue as a viable entity Addresses the company’s critical business functions to maintain operations Implements alterative processes or systems, until normal systems availability is returned Involves preparations for a disaster and addresses the procedures to re-establish system after a loss Specific to Restoring the existing systems to normal availability Executed in parallel to the Business Continuity Plan 10

Types of Disaster Recovery Plans Emergency Plan Actions to be undertaken when a disaster occurs Identify situations which require the plan to be invoked Data Back Up Plan Specifies the type of backups to be kept, frequency of back ups, procedures locations, personal, priorities and time frame Needs continuous updates as changes occur Recovery/Restore Plan Test Plan Procedures to restore full information systems capabilities Formation of recovery committees, responsibilities, and guidelines Very plans and activities will functions as intended Identify deficiencies in the emergency, back up and recovery plans Actions, and organization Confirmation of successful restorations NOTE: Disaster Recovery is integral to a Business Continuity Program but is focused on restoring normal systems operation Well maintained and tested Data Back up and Restoration Plans are key to Resilience and Recovery 11

Five Levels of Testing Business Continuity and Disaster Plans Document Review Walkthrough Simulation Review of recovery , operations, resumption plans and procedures Perform by individuals Provide feedback to document owners Performed by teams Process maps of recovery , operations, resumption plans and procedures Brainstorm and discussion brings out new issues, ideas Provide feedback to document owners Walkthrough of recover, operations, resumption plans and procedures in a scripted “case study” or ‘scenario” Performed by teams Places participants in a mock disaster setting that helps discern real issues more easily Full or partial workload is applied to recovery systems Performed by teams Test actual system readiness and accuracy of procedures Production systems continue to operate and support actual business processes Parallel Test Cutover Test Production system are shut down or disconnect; recover system assume full actual workload Risk of interrupting a real business challenges the Disaster Recovery system 12

Business Continuity Plan Should be prepared and available for reference throughout any crisis. Invoked according to the instructions contained in the document. May be a manual/paper based method or an alternative electronic solution Subject to independent business/technical SME review and approval Interrelated with the Disaster Recovery Plans, and Data Back up & Restore instructions Test the alternative business process in advance according to the procedures specified in the implementation protocols. Include executable implementation protocols, as appropriate. If deviations are encountered, these should be resolved and appended to the results. Should be periodically reviewed to keep it aligned with business imperatives and all references to roles/ personnel should be kept up to date. . A copy of the Business Continuity Plan should be held in a secure location that can be accessed in the event of a disaster 13

Decision to Implement an Alternative Business Process Only one role should be accountable for a process. For business continuity management, the ultimate accountability lies with the Process Owner with support from the Crisis Committee. 14

Business Continuity Planning and Implementation Relies on Strong Interactions Amongst Several Areas of the Organization System Support ( IT) Operations Leadership/ Process owners Quality System Owners 15

Business Continuity Execution Process Serious Event, Emergency or Disaster Follow the documented plan or develop a revised plan if the criteria of the crisis differs from the plan Business Continuity closeout and summary report Reconciliation of data with the normal operations system When the crisis is over implement the transition to the normal business process again following change management. Establish communications, monitoring and regular status reporting Implement the plan, following change management Ensure all applicable personnel are trained on the alternative process Manufacturing continues under alternative process

Communication is Critical Many parties need to know the condition of the organization Employees, supplier, customers, regulators, authorities, share holders, community Method of communication Telephone call trees, email/text distribution lists , website, signage, media Alternative means of communication must be identified Topics to communicate Prioritization of process to restore if multiple process are impacted Prioritization of system restoration Method and chain of issue escalation Status of alternative process and steps to restore normal operations 17

Business Continuity Plan Content : Potential serious incident/emergency scenarios Business Continuity strategies, including: selection of alternative strategies specification of business process Recovery Point Objective (RPO) - the point in time to which data must be restored following a failure or disaster loss, e.g., a restore to the previous night's backup will not include today's transactions - is this acceptable? Recovery Time Objective (RTO) - the time within which the business process must be restored following a failure or disaster. Business Continuity Plan Process ABC Revision Date – 01-Jan-2018 immediate steps to be taken to minimize further impact interim processes required to manage disruptions required personnel and responsibilities, including: Key Contacts, e.g., Process Owner, System Owner, Suppliers, Quality Unit should be listed with their contact details. Crisis Committee criteria for reporting to the Crisis Committee representatives: Chair (e.g., Process Owner) representatives from areas of the business, such as Environmental, Health and Safety, Security, Utilities, Engineering, HR/Communications, Production, IT, Laboratories, Facilities, etc. criteria for a crisis and invoking the BCP stage timings and escalation process progress tracking communication of the crisis situation and progress reporting 18

Implement the Plan Applicable to the Crisis Business Continuity and Disaster Recovery options should be chosen based on risk, and consider the criticality and complexity of the system. Low Impact System Medium Impact System High Impact System Disaster Recovery DR Plan, DR Instructions (Back up and Restore) available, Disaster Recovery test frequency and type based on risk DR Plan, DR Instructions available, DR Test protocols available, Disaster Recovery test frequency and type based on risk DR Plan, DR Instructions available, DR Test protocols available, Disaster Recovery test frequency and type based on risk Alternative Process Options Alternative work locations Alternate personnel Communications Emergency , support of business process Stand-by Assists and equipment Access to procedures and business records Business Continuity Management No action System included in BCP System included in BCP, Event/Disaster Scenarios Considered, Alternative Options Considered Restoration and Recovery Options Repairs to facilities, equipment Replacement of equipment Restoration of Data backups Confirmation of impact to Data created prior to the incident Resumption of normal operations 19

Ensure All Applicable Personal are Trained Normal operations and procedures Emergency procedures Alternative process procedures – confirm training when invoking the specific alternative process Recovery procedures Resumption procedures 20

Manufacturing Continues Under Alternative Process Incidents Reports and Change Management should document the prioritized Alternative Processes being followed and which records are impacted. Determine what data is required to continue the critical business process and how can it be obtained Options to access the current data for in-process work Restore System Backups Paper printouts of the original source system, before it went offline Secondary systems that store the same information Direct queries to the database if the application server front end is not available Options to create and process new data for the work during the alternative process Paper forms and templates Additional 2nd person verification Alternative faculties/equipment Repurpose the training environment 21

Transition to the Normal Process As systems are recovered following successful disaster recovery, the return to normal operations must be carefully controlled. Not all systems may be returned to normal service at the same time . This will require careful collaboration amongst the interfaced processes and system. Much like the transition to alternative process decisions must be made on current in-process activities and new work to resume under normal operations. Continue to use Change Management to document and authorize the transition to normal operations. 22

Reconciliation of Data with Normal Operations Systems Data created outside the normal process must be reconciled Determine the retention locations for any data/records created by the alternative processes. Do the systems used for Normal Operations need to update with the alternative records? If the paper data is copied/scanned into the normal operations systems ensure a true copy process is followed retain the metadata and audit trail required to ensure that the full meaning of the data are kept and its history may be reconstructed. If electronic data are transferred to the normal operations systems, validation should include checks that data are not altered in value and/or meaning during this migration process 23

Summarize the Closure of the Business Continuity Actions BCP/DR Summary Report A communication to the Business should be sent out immediately after the successful resolution of the disaster and resumption of normal service. Following the return to normal business: Results of the BCP, BCP protocols, and/or DR protocols should be reported. Include all findings and deviations, lessons learned, and recommended changes to the BCP Include any ongoing monitoring or CAPAs as a result of the event Communicate the BCP/DR Summary Report to affected stakeholders. 24

Maintaining Business Continuity and Disaster Recovery Plans Events that require review and modification of Business Continuity and Disaster Recovery Plans: Changes in: Business process and procedures IT systems and application ( upgrades, replacements, new or additional applications) IT Architecture ( network, Data centers, application servers) Services provides Organizational structure Lesson learned from BCP implementation Re-test the BCP/DR to ensure that the changes work as intended and do not cause unexpected affects elsewhere 25

Case Study 26

Case study 1: Failure scenario: Sterile filling line consisting of a Sterility Isolator and Filling control systems. Both SCADA controlled (Windows 7 Industrial PCs), both PLC controlled (mechanical control and alarm processing back to the SCADA control Unit). System is validated to keep running on the PLCs when the SCADA PCs are offline. The SCADA units go offline (both), and the line keeps running a batch of a life saving product nearing market stock-out. The SCADA units will both require re-installation. Both SCADA units also send the process data to a historian via an OPC connection. Both continuous and batch data is sent to the historian and a batch end report summarizes the batch, any critical alarms, including CPP excursions. e.g., Fill volume, fill line pressure (both will trigger an alarm in the event of an excursion from validated limits in the PLC and stop the line). The line does not stop during the batch and filling is successfully completed. What is the first thing the site needs to have to keep the line running? Q. What will be the impact to the electronic data? Q. Given the data impact– can you justify the running of the machine without the SCADA units? Q. What design change could be made to ensure the historian data is available? Q. 27

Case study 1: Answers Q. What is the first thing the site needs to have to keep the line running? R. A documented Business Continuity Plan based on the system validation, to keep running until there is a critical alarm . Q. What be the impact on the electronic data? A. All data generated after the last backup on the SCADA unit will not be electronically captured, such as Login/out logs, operational logs include interventions, alarms from the PLC. All data from the historian from the point of failure of the SCADA units. R. Given the data impact – can you justify the running of the machine without the SCADA units? A. The PLCs are validated to run without the SCADA units and it has been proven that a critical alarm will stop the line. (Cannot re-start without the SCADA units) S. What design change could be made to ensure the historian data is available? A. OPC connection from the PLC unit instead of the SCADA. 28

Case Study 2 Failure scenario: Building management system (GMP) that record temperature, pressures, humidity across the site in manufacturing, packaging, warehouses, cold storage, freezers. The system architecture is such that it can store data in the various hardware controllers if the server is unavailable for up to 72hrs. This has been validated for all controllers based on the number of instrument inputs. The server, even though virtual and mirrored has become unavailable and will require a “bare metal” restore including data. The HVAC control is unaffected by the loss of the server (this has been validated). Q. What is the first thing the site needs to have to protect raw materials, in-process and finished product? Q. What manual approach might be necessary as a precautionary measure before the server is rebuilt? Q. What is the typical minimum frequency that might be deemed acceptable to record values? Q. What will be the electronic data impact? Q. Given the data impact– how could this be mitigated against during operations? Q. What is the recovery time objective for the BMS server? 29

Case Study 2 - Answer Q. What is the first thing the site needs to have to protect raw materials, in-process and finished product? A. A documented Business Continuity Plan Q. What manual approach might be necessary as a precautionary measure before the server is rebuilt? B. An approved SOP driven approach to manually recording the critical parameters from the BMS control panels at certain intervals. Q. What is the typical minimum frequency that might be deemed acceptable to record values? A. Depends on validation studies for maximum length of excursions for environmental limits. Q. What will be the electronic data impact? C. There may be a data gap between the last backup and the first buffered data in the BMS controllers recorded that had not been sent to the server after it failed. Q. Given the data impact – how could this be mitigated against during operations? D. By performing a perioidic alarm check on the BMS each day (documented) – this may minimise the period for which no data or alarms were reviewed. Q. What is the recovery time objective for the BMS server? A. Less than 72 Hours to avoid over-writing of data in the buffers on the controllers. 30

Summary A serious incident or disaster can happen at anytime based on any number of causes. It is essential than an organization is able to recover to an operational state within a reasonably short period. It is not a matter of IF but WHEN Identify the Critical Business processes that must continue and the data required as part of the process Business continuity includes three key elements Resilience Understanding of the Process and Systems Interfaces IT Architecture, Remote Data Centers, Cloud System Redundancy and spare capacity Recovery Disaster Recovery Plans Well maintained Back up programs Tested restoration protocols Contingency Leadership teams Escalation and Communication Predefined Alternative Process Secured copies of procedures , forms and templates 31

Q&A 32