Automation for System Safety Analysis Jane T. Malin, Principal

Automation for System Safety Analysis Jane T. Malin, Principal Investigator Project: Automated Tool and Method for System Safety Analysis Software Assurance Symposium September, 2008 Complex systems typically fail because of the unintended consequences of their design, the things they do that were not intended to be done. - M. Griffin, System Engineering and the “Two Cultures” of Engineering, March 28, 2007 1 SAS 08 Automation for System Safety Analysis Malin

Overview Problem/NASA Relevance Technical Approach and Overview 2008 Target Capability Crew Exploration Vehicle (CEV) Launch Abort System Case Data – Constellation (Cx) failure modes and effects analysis/critical items lists (FMEA/CILs) Technical Challenges – Information Extraction – Semi-Automated Model Construction – Analysis and Test Case Generation 2009 Planned Capability Potential Applications 2 SAS 08 Automation for System Safety Analysis Malin

Problem and NASA Relevance NASA needs early evaluation of software (SW) requirements and design, to reduce softwaresystem integration risks – Assess system failures and anomalous conditions that may challenge software in system integration testing – Identify robustness issues early (and often) – Identify requirements gaps early (and often) Project test case: NASA Constellation (Cx) Launch Abort System (LAS) for Pad Abort PLANT and Environment FAULTS/Reliability ‘Activate’ Faults and Influence Failures ‘Activate’ Faults and Influence Failures Operations and Stresses SOFTWARE FAULTS/Reliability 3 SAS 08 Automation for System Safety Analysis Malin

Technical Approach Systematic semi-automated extraction and analysis for early evaluation and rapid update – Capture model of the controlled system architecture Abstract physical architecture models with subsystems, functions, interfaces, connections Extract directly from requirements and design text and data – Capture risks and hazards in model Constraints, hazards, risks from requirements and design Risk and failure libraries – Analyze and simulate to identify risks and constraints Analyze and simulate hazard/risk propagation in the system Use operational and off-nominal scenarios and configurations – Identify possible test scenarios for virtual system integration testing 4 SAS 08 Automation for System Safety Analysis Malin

Technology Overview Hazard Identification Tool (HIT) Aerospace Ontology CONFIG Hybrid Simulation Extract and Model Information Extractor Identify Test Cases Requirements Text Analyze, Simulate and Test Early - Identify interaction-propagation paths - Investigate influence of timing - Perform Virtual Tests Virtual System Integration Laboratory (VSIL) 5 SAS 08 Automation for System Safety Analysis Malin

Modeler: Architecture Model and Visualization of a Set of Requirements [C.1] Telecommunication Subsystem (TeleSub) [C.1.1] The CDHC sends the TeleSub a compressed picture. [FG.1] [TeleSub C.1.4] [C.1.2] The CDHC sends the TeleSub telemetry. [FG.2] [FR.1] [FR.5] [TeleSub C.1.5] [C.1.3] The CDHC sends In View of Ground alerts to the TeleSub. [DP.5.6] [TeleSub C.1.6] [C.1.4] The CDHC receives plan files from the TeleSub. [FR.3] [TeleSub C.1.3] [C.1.5] The CDHC receives ground commands from the TeleSub. [FR.3] [TeleSub C.1.2] [C.1.6] The CDHC receives the TeleSub operating state from the TeleSub. [DP.5.5] [TeleSub C.1.1] [C.2] Camera Subsystem [C.2.1] The CDHC sends the Camera a "take picture" command. [FG.2] [FR.1] [FR.3] [C.2.2] The CDHC sends the Camera x, y and z gimballing coordinates. [FG.2] [FR.1] [FR.3] [C.2.3] The CDHC sends a turn on command to the Camera. [DP.5.3] [H Constraint 1.1.4] [C.2.4] The CDHC sends a turn off command to the Camera. [DP.5.3] [C.2.5] The CDHC receives a compressed picture file from the Camera. [FG.1] [FG.2] [FR.1] [C.4] Attitude Determination Subsystem (ADS) [C.4.1] The CDHC receives an In View of Ground alert from the ADS. [DP.5.6] [ADS] [C.4.2] The CDHC receives the ADS operating state from the ADS. [DP.5.5] [ADS] Note: CDHC is Command and Data Handling Computer Physical/Functional Architecture Visualization 6 SAS 08 Automation for System Safety Analysis Malin

CONFIG Simulation: Assess Timed Scenarios CONFIG simulation tool used for software virtual validation testing for NASA 1997 90-day manned Lunar Life Support Test Software: Intelligent control for gas storage and transfer Models: Gas volumes and processing systems controlled by software; mixed fidelity, discrete and continuous Testing: Simulated failures and imbalances that would not be tested in hardware-software integration Too slow to develop, too expensive, too destructive Results: Identified software requirements deficiency due to unintended consequences of integrating gas processing systems 7 SAS 08 Automation for System Safety Analysis Malin

Virtual System Integration Lab (VSIL) Triakis has used VSIL in 25 avionics verification projects Project Output to VSIL: Models and test definitions Models and Test Definitions DE: detailed executable, the simulation of the embedded controller hardware RAM/ROM: memories ES: executable specifications I/O: input/output V&V: verification and validation CPU: processor 8 SAS 08 Automation for System Safety Analysis Malin

2008 Target Capability Integration: Information extraction, architecture modeling and test generation – Model parts extracted from requirements and FMEA/CIL texts XML output, including reference traces Components, physical hierarchy, connections, interface components, flows/resources, time or phase context Functions, vulnerabilities, limits, failures, causes – Ontology for model extraction and semi-automated modeling Identify types of components, functions, problems, resources Paths: A provides power to B; C receives command data from B Functions and failures: B processes command data; B failure mode is No Output command to D; cause of no output is B does not receive power. – Semi-automated model development from extracted model parts Component model library: Resource producer; Data processor Generic functions, failures and influences: Resource problem, Stressor, Data rate problem, Data Integrity problem – Model visualization for overview and completeness checking – Simulation and path analysis to identify hazardous configurations, scenarios and test cases Where failure or degradation of required functions results from unintended system interactions Project Participants – CEV Flight Software Engineering, Abort Decision Logic, Abort Sequence – Orion Software Safety and Mission Assurance 9 SAS 08 Automation for System Safety Analysis Malin

CEV Launch Abort System (LAS) Case Crew Exploration Vehicle (CEV) Pad Abort Sequence 10 notional SAS 08 Automation for System Safety Analysis Malin

CEV Launch Abort System Case CEV Crew Module (CM) software controls the Pad Abort Sequence – LAS events, trajectory – No direct command feedback Components for the case – Paths and interactions for commands firing separation pyros during pad abort sequence CM computer Remote Interface Unit LAS Pyros Possible Addition Case: Inertial measurement unit (IMU) GN&C Abort motor – Summer Systems Engineering Intern manually built TEAMS models of pyros and Remote Interface Unit (RIU) from FMEA/CILs Need to perform simulation rather than pure path analysis on the LAS case – Timing is important in aborts – Hazard Identification Tool (HIT) path analysis models do not capture timing – CONFIG simulations can use timing 11 SAS 08 Automation for System Safety Analysis Malin

Data – ConOps, Requirements, Safety Analyses Developing tools and methods using documents and data from CEV sources, both NASA and Orion contractor Met with NASA expert on Orion software that controls Launch Aborts – Identified key CEV documents and confirmed analysis approach Orion Contractor’s Concept of Operations – Best guess at the Abort Sequence Interface Requirements Documents Interface Control Documents, when they become available Project Orion Flight System Safety Hazard Analysis FMEA/CILs (preliminary now) – Determined that much key information (sensors, feedback) is TBD, and being defined by the Cx Integrated Abort Team 12 SAS 08 Automation for System Safety Analysis Malin

Technical Challenges Limitations of early life-cycle requirements, design, hazard analysis and FMEA/CIL as sources for – Automatic extraction of model information from requirements and design text – Semi-automatic construction of models from extracted information, for simulation and visualization Combining graph analysis and simulation to identify possible hazard paths and off-nominal test scenarios for complex system interaction models Maturation Challenge: Develop mature software prototypes that can be used to develop products for broader use 13 SAS 08 Automation for System Safety Analysis Malin

Information Extraction Objective: Extract information from CEV sources for semi-automated model construction Information Extraction Evaluation – Success Criterion: % of available model information extracted, compared to % of model information available Types of Extractions (from text to XML) – Interface Requirements Components, connections, flows/resources, reference trace Some interface components, vulnerabilities, functions, limits, context (time or phase) – FMEA/CILs Components, system to component hierarchy, interfaces, subcomponents, functions, failures, causes, effects – Architecture Descriptions Components, interfaces, hierarchy, functions; some design parameters, acronyms Challenges – Multiple document formats require definition of data structures for each document, some difficult sentence parsing – Indirect access to Cradle requirements via PDF documents Progress: – Parser improved by incorporating NESC-funded parser from Univ. Central Florida – Experience coding multiple document data structures, leading to format specification approach – Successful extraction from Cradle-based PDF docs. 14 SAS 08 Automation for System Safety Analysis Malin

Example Information Extraction Benchmarking LAS Description: The LAS consists of a nose cone, a canard section which enables the LAS to reorient the CM for parachute deployment following an abort, three propulsive motors (attitude control, jettison, and abort), a bi-conic adapter which provides the structural interface to the CM, and a boost protective cover (BPC) sized for ascent heating to protect CM thermal protection system (TPS) coatings. Ideal Model Extraction Benchmark: Top Level: LAS Function: reorient the CM Agent: LAS Action: reorient Operand: CM Function: control attitude Agent: LAS? Action: control Operand: ? Variable: attitude Function: deploy parachute Agent: LAS? Action: deploy Operand: parachute LAS Components Component: Nose cone Component: Canard Section Function: ? Component Enables Function: LAS: reorient the CM Function Enables Function: LAS?: deploy parachute (following an abort) Component Group: Three propulsive motors Component: attitude control motor Enables Function: LAS?: control attitude Component: A bi-conic adapter Connection Connector: bi-conic adapter From: LAS To: CM Type: Structural Component: Boost protective cover Acronym: Boost protective cover BPC Design parameter: size Determined by: ascent heating Function: protect CM thermal protection system coatings Agent: Boost protective cover Action: protect Operand: CM thermal protection system coatings Other: CM Component: Thermal protection system Acronym: thermal protection system TPS 15 SAS 08 Automation for System Safety Analysis Malin

Model Construction Objective: Use extracted model information for semi-automatic model construction Challenges – LAS case with timing issues requires CONFIG simulation, not just HIT architecture model – Missing information in pre-PDR documents Operating modes, vulnerabilities, side effects etc. Progress – CONFIG provides visualization and supports libraries of generic components – Concept for use of Aerospace Ontology hierarchies with CONFIG library for generic components, operating modes, functions, side effects, problems 16 SAS 08 Automation for System Safety Analysis Malin

Analysis and Test Case Generation Objectives: – Identify and evaluate failure and hazard propagation in the system model Unintended system interactions and unanalyzed propagation of failure effects – Generate corresponding off-nominal test cases (configurations, scenarios) Evaluations – Hazard Analysis Success Criterion: Are new hazards and failures identified, compared to standard method? – Test Case Success Criterion: Does model-based hazards and failures analysis make test generation easier than current methods? Challenges – Path analysis algorithm for HIT models needs to be adapted, because LAS case will use CONFIG simulation rather than HIT – Extracting information for operational and failure scenarios has not yet been addressed FMEA/CIL effects information can provide parts of failure scenarios Progress – Concept for combining HIT and CONFIG models, using CONFIG Inner Models for subsystem details 17 SAS 08 Automation for System Safety Analysis Malin

2009 Planned Capability Capabilities should be valuable from pre-PDR through operations Continue tool enhancements focusing on – Off-nominal test scenario discovery and evaluation – Component model library and generic defaults Use on a new CEV case – more complex interactions and more complete system information Deliver – Tool prototype files – Information extraction tool, Aerospace ontology, HIT graph modeling and analysis, CONFIG simulation modeling, model libraries – Documentation – methods, tools, user manuals 18 SAS 08 Automation for System Safety Analysis Malin

Future Applications Improve efficiency and repeatability of system and software risk analysis – Reduce time spent reanalyzing when specifications and designs change Visualize integrated requirements – Combined success and failure spaces – Combined system and operation/event spaces Validate requirements and perform integration tests early with low-fidelity and multi-fidelity simulation Validate FMEAs and fault trees Evaluate completeness and consistency of requirements and risk – Support requirements traceability evaluations – Enhance analysis with reliability and event probability information 19 SAS 08 Automation for System Safety Analysis Malin