A Scorecard for Cyber Resilience: What We Have Observed (Partial

A Scorecard for Cyber Resilience: What We Have Observed (Partial Version) Robert A. Vrtis, CISSP Senior Engineer, CERT Software Engineering Institute Carnegie Mellon University Andrew F. Hoover, CISA, CRISC, CISSP Senior Engineer, CERT Software Engineering Institute Carnegie Mellon University [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority.

Copyright 2016 Carnegie Mellon University (CMU) This material is based upon work funded and supported by Department of Homeland Security under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute (SEI), a federally funded research and development center sponsored by the United States Department of Defense. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of Department of Homeland Security or the United States Department of Defense. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-0925) or higher DoD authority. Carnegie Mellon and CERT are registered marks of Carnegie Mellon University. DM-0003936 [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Agenda What Is the Cyber Resilience Review (CRR)? How CRR Data Is Collected Overview of CRR Data that Has Been Collected CRR Data Analysis Key Observations – Maturity Indicator Level Practice Observations – Domain-Specific Observations NIST Cybersecurity Framework Observations [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

What Do We Mean by Resilience? [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Operational resilience: The emergent property of an organization that can continue to carry out its mission after disruption that does not exceed its operational limit ―CERT-RMM Where does the disruption come from? Realized risk. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

What Is the Cyber Resilience Review (CRR)? A U.S. Department of Homeland Security (DHS) initiative intended to help the nation’s critical infrastructure providers understand their operational resilience and ability to manage cyber risk A review of the overall health of an organization’s cybersecurity program, as it relates to a specific critical service A tool that allows an organization to – develop an understanding of process-based cybersecurity capabilities – improve its ability to manage cyber risk to its critical services and related assets – compare its capabilities to the criteria of the National Institute of Standards and Technology (NIST) Cybersecurity Framework (CSF) [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Overview of the CRR The CRR is derived from the CERT Resilience Management Model (CERT-RMM). It is a structured assessment conducted during a one day, facilitated session. The CRR session is facilitated by multiple navigators (DHS and CERT) who solicit the answers to 297 questions. The CRR results in a summary report that provides the organization with suggested options for consideration. All CRR practices have been mapped to the subcategories of the NIST CSF, allowing an organization to assess it’s capabilities relative to the CSF criteria. This analysis was conducted using the CRR results of 245 organizations. [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Overview of the NIST Cybersecurity Framework (CSF) Recognizing that the national and economic security of the United States depends on the reliable functioning of critical infrastructure, the President issued Executive Order (EO) 13636, Improving Critical Infrastructure Cybersecurity, in February 2013. The order directed NIST to work with stakeholders to develop a voluntary framework for reducing cyber risks to critical infrastructure. Created through collaboration between industry and government, the framework consists of standards, guidelines, and practices, divided into five functional areas, to promote the protection of critical infrastructure. [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

CSF – CRR Mapping The CRR enables an organization to assess its capabilities relative to CSF Maps the CSF categories and subcategories to the CRR goals and practices. This mapping can be found in the DHS C3 Voluntary Program website. [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Who Participates? Chemical Food and Agriculture Commercial Facilities Government Facilities Communications Critical Manufacturing Health Care and Public Health Dams Information Technology Defense Industrial Base Nuclear Reactors, Materials, and Waste Emergency Services Transportation Systems Energy Water Financial Services Organizations within Critical Infrastructure and Key Resources (CIKR) sectors State, Local, Tribal, and Territorial (SLTT) governments, within the United States (and its territories) Participation is voluntary and protected by the Protected Critical Infrastructure Information (PCII) Program [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Service-Oriented Approach The CRR has a service focus. An organization deploys its assets (people, information, technology, and facilities) to support specific operational missions, or services. For example, the wastewater processing service in a water treatment plant. A service orientation enables the identification of assets important to achieving an organizational or sector mission. The service is used to scope the CRR. [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

CRR Domains―What We Examine CRR Domains Asset Management Controls Management Configuration and Change Management Vulnerability Management Incident Management Service Continuity Management Based on the CERT-RMM The ten domains in the CRR represent important areas that contribute to the cyber resilience of an organization. The domains focus on practices an organization should have in place to assure the protection and sustainment of its critical service. Risk Management External Dependencies Management Training and Awareness Situational Awareness [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

CRR Architecture Overview [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. What to Do Making it Stick Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Cyber Resilience Review Numbers Number of Goals Number of Goal Practices Number of MIL Practices Asset Management 7 24 13 Controls Management 4 7 13 Configuration and Change Management 3 15 13 Vulnerability Management 4 12 13 Incident Management 5 23 13 Service Continuity Management 4 15 13 Risk Management 5 13 13 External Dependencies Management 5 14 13 Training and Awareness 2 8 13 Situational Awareness 3 8 13 167 130 CRR Domains Total [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Maturity Indicator Level (MIL) Scale The MIL scale measures process institutionalization Higher degrees of institutionalization translate to more stable processes that – produce consistent results over time – are retained during times of stress MIL Level 5 – Defined Processes are acculturated, defined, measured, and governed MIL Level 4 – Measured MIL Level 3 – Managed MIL Level 2 – Planned Practices are performed MIL Level 1 – Performed Practices are incompletely performed, or not performed MIL Level 0 – Incomplete [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Agenda What Is the Cyber Resilience Review (CRR)? How CRR Data Is Collected Overview of CRR Data that Has Been Collected CRR Data Analysis Key Observations – Maturity Indicator Level Practice Observations – Domain-Specific Observations NIST Cybersecurity Framework Observations [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Conducting the CRR: Role of the Navigator Navigators work with the organization to ensure that the appropriate subject matter experts participate. Two navigators execute the CRR with the organization’s representatives. Navigators collect data independently. Following the CRR, the navigators reconcile discrepancies and validate the data. Navigators then work with the organization to review the initial draft report. [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

CRR Data Capture Form and Report Preparation [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Each Domain is in its own section. Each Domain is divided into goals. Each practice question has 3 possible answers: Yes Incomplete No Each practice question has imbedded guidance. Following the completion of the CRR, each navigator’s answers will be compared reconciled An initial report is then generated. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

CRR Scoring Depictions [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Contact Information This is a partial version of the presentation. Please contact us for the full version that includes data and observations. Robert A. Vrtis, CISSP Senior Engineer Cybersecurity Assurance - CS2 CERT Software Engineering Institute Carnegie Mellon University [email protected] Andrew F. Hoover, CISA, CRISC, CISSP Senior Engineer Cybersecurity Assurance – CS2 CERT Software Engineering Institute Carnegie Mellon University [email protected] [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

Supporting Material [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.

List of Acronyms Used AM – Asset Management CCM – Configuration and Change Management CIKR – Critical Infrastructure and Key Resources CM – Controls Management CMU – Carnegie Mellon University CRR - Cyber Resilience Review DHS – Department of Homeland Security EDM - External Dependencies Management IM – Incident Management MIL – Maturity Indicator Level RM – Risk Management RMM – Resilience Management Model SA – Situational Awareness SCM – Service Continuity Management SEI – Software Engineering Institute SLTT – State, Local, Tribal, and Territorial TA – Training and Awareness VM – Vulnerability Management [DISTRIBUTION STATEMENT F] Further dissemination only as directed by Department of Homeland Security (2015-09-25) or higher DoD authority. Copyright 2016 Information Systems Audit and Control Association, Inc. All rights reserved.