IV&V Facility NASA Software IV&V Facility Metrics Data Program

IV&V Facility NASA Software IV&V Facility Metrics Data Program Mike Chapman Pat Callis Ken McGill June 5, 2003 [email protected] 304-367-8341 [email protected] 304-367-8309 [email protected] 304-367-8300 1

IV&V Facility The Metrics Data Program Goal Establish a centralized repository that provides consistent, fully-involved software product data across multiple domains: To improve the effectiveness of software assurance To improve the effectiveness of software research To improve the ability of projects to predict software errors early in the lifecycle 2

Data, Data Everywhere: The Data Drought IV&V Facility There is very little fully-involved software product data Error data associated to smallest functional unit Requirements traced through the design to the smallest functional unit available to those who need it. 3

IV&V Facility SARP PI Survey To what degree has a lack of software defect data from actual NASA projects impacted your SARP funded research? - Greatly (9). Lack of software defect data from actual NASA project has seriously hampered my research - Moderately (8). It would be nice to have more (or some) real project data, but I have found non-NASA data sources or other workarounds. - Not significantly (5). My project has not been impacted because I either have all the data I need or I don't need software defect data Note: The totals are in parentheses. 4

The Lack of Data from a Project Perspective IV&V Facility Error data associated to smallest functional unit There is little value in this activity for the project Requirements traced through the design to the smallest functional unit Projects only need to trace requirements to the executable (CSC level or above) Fully-involved data made available to the research community Vulnerability of the program Proprietary issues 5

The Quest for the Holy Grail IV&V Facility Regardless of the development model the following is needed: Requirements Traceability Design Code (Many to one issue.) Problem Reports dev, test, user, maintenance Association to the smallest functional unit. (smallest functional unit) 6

Recruitment of Project Data IV&V Facility Existing repository data Identifier Language LOC Domain Error Data JM-1 C 315K Real-Time 8 years KC C 750K Data System 5 years CM-1 C 20K Instrument 2 years CM-4 C 40K Flight N/A Error Data Enhancement Program 7

Error Data Enhancement Program IV&V Facility The goal of the enhancement effort is to successfully recruit projects to work with the MDP to provide fullyinvolved software product data – the Holy Grail. The MDP team will provide: Requirements analysis and traceability support Configuration Management Support (Error Tracking) Metrics generation and analysis Database and web support Machine learning analysis 8

MDP Repository IV&V Facility Project Error Data Other Project Data MDP Web Site ClearQuest Interface Queries Sanitized Data Firewall Stored Metrics and Error Data ClearQuest Sanitized Data Project Security Project1 .Project n Project Metrics and Error Data 9

Benefits IV&V Facility Agency benefits: The improved ability to predict error early in the lifecycle The improved ability to assess the quality of the software The research community benefits: Availability of quality error and metric data Availability of a support team for data needs Participating projects benefit: Additional metrics analysis Additional error analysis Problem tracking tool Other support such as requirements traceability 10

Where to Find More Information IV&V Facility mdp.ivv.nasa.gov 11

Site Metrics IV&V Facility Web Site activity for 3 months: 596 hits 46 accounts 146 logins 85 downloads of data 12

Special Data Requests IV&V Facility Five time stamps of KC-2 data Sanitized activity fields of JM-1 data Error Reports from CSCI of KC Note: Five papers have been written from the repository data so far. 13

Current Data Request IV&V Facility CM-1 requirements data and associated errors - JPL CM-1 data including metrics – Dolores Wallace SATC KC semantic metrics generation – Letha Etzkorn UA – Huntsville JM-1 time stamp data (five sets) – Tim Menzies WVU 14

A Study IV&V Facility Metric Combination % of Errors % of Code Probability of Failure (%) l 0.047015525 50% 34% 71% loc 9.197AND l 0.066 30% 20% 37% Metric Combination Probability of Module Error Probability of Failure (%) % of Severity One Errors v(g) 45.9 & e 1340053 80% 0% 50% (8/16) v(g) 45.9 & loc 150 73% 0% 33% (23/69) v(G) – cyclomatic complexity – independent linear paths ev(G) – essential complexity – unstructured constructs e – programming effort – mental effort l – (2/u1)*(u2/N2)-program level – the level at which a program can be understood 15

IV&V Facility Background Slides on Metrics 16

McCabe Metrics IV&V Facility Halstead: programmers read code. Too many “words” error Mccabe: paths between “words” Twisted paths error v(G): cyclomatic complexity # path(ish)s edges-nodes 1 m # one entry/one exit sub-graphs ev(G): essential complexity v(G) – m iv(G): design complexity (reflects complexity17 of calls to other modules)

Halstead Metrics IV&V Facility µ µ1 µ2 N length N1 N2 V volume N*log2(µ) V’ (2 µ2’)*log2(2 µ2’) L D L’ E T level V’/V difficulty 1/L 1/D effort V/L’ time E/18 e.g. 2 2 3 N1 3 N2 2 µ1 2 µ2 2 µ1’ 2(ish) µ2’ #input parameters µ1 N2 µ2 N1 Could be found via simple tokenizers 18

Operators IV&V Facility ! % & * , - ./ ; ? ! - * / % & && -- - return size of enum struct delete continue new break union if ( ) if ( ) else switch ( ) default: case label : goto label do while ( ) while ( ) do for ( ; ; ) this- [] {} function name ( ) ( ) in any other cases not covered ‘?’ ‘ ‘: 19

Operands IV&V Facility Variables and identifiers Constants (numeric literal/string) Function names when used during calls 20

Error Metrics IV&V Facility Module LOC ER ED* 6 mth 1 yr 2 yr CSC1/mod1 59 6 101.7 3 4 5 CSC2/mod1 190 6 31.58 2 4 5 CSC1/mod9 40 5 125 1 2 3 CSC7/mod27 47 3 63.83 CSC4/mod10 287 2 6.9 1 2 2 CSC6/mod51 482 10 20.8 1 4 *ED ER/KLOC 21

OO Metrics IV&V Facility Number of Children (NOC) – number of sub-classes Depth – level of class in the class hierarchy Response for Class (RFC) – number of local methods plus the number of methods called by local methods ( 100) Weighted Methods per Class (WMC) – sum of the complexities of the methods ( 100) Coupling Between Object Classes (CBO) – dependency on classes outside the class hierarchy ( 5) Lack of Cohesion of Methods (LOCM) – the use of local instance variable by local methods 22

ARM Metrics IV&V Facility Weak Phrases (adequate, be able to) – clauses that cause uncertainty Incomplete (TBD, TBR) – Words and phrases that indicate the spec may not be fully developed Options (can, optionally) – Words that give the developer latitude Imperatives (shall, may, will, should) – Words that are explicit Continuances (below, as follows, and) Extensive use of continuances can indicate complex requirements. Directives (for example, figure, table) – examples or illustrations Lines of text (size) Document structure (levels) 23

IV&V Facility Program Background Slides 24

Problem Report Fields IV&V Facility Error Identifier: (Alpha-numeric) Headline: (text- short description) Submitted-on: (Date yymmdd) Severity: (1 thru 5) Status: (NVTCDARMB) System Mode: (operations versus test) Request type: (Problem or enhancement) Problem-type: (requirements, design, source code, COTS, documentation, hardware,etc) Problem Mode: (PR or Action Item) Assigned-to: CCB Approval: (Date) Impacts-csci: (high level design element) Impacts-csc: CSC Impacts-class/file: class/file Impacts-method/function/module: Method/function/module Impacts – Requirement Impacts – design element Resolution: source code, COTS, documentation, not a bug, unreproducible Problem: (text) Analysis: (text) Resolution: (text) Closed-on: (Date) 25

IV&V Facility Project Non-Specific Universal Error Identifier: (Alpha-numeric)* Submitted-on: (yymmdd) Severity: (1 thru 5) Status: (NVTCDARMB) Mode: (operations versus test) Request type: (Problem or enhancement) Problem-type: (requirements, design, source code, COTS, documentation,etc) Impacts-csci: (high level software design element)* Documents: (What documents are affected?) CCB Approval: (Date) Resolution: source code, COTS, documentation, not a bug, unreproducible Verified-on: (Date) Closed-on: (Date) * May need sanitized. 26

IV&V Facility Project Non-specific Expanded Impact data: Impacts-csc: CSC* Impacts-class/file: class/file* Impacts-method/function/module: Method/function/module* Recommend-change: source code Process data: How-found: (e.g., Acceptance Test) When-found: (e.g., Acceptance Testing) Analysis-due: 020322 Assigned-Eval-on: 020322 Assigned-Implement-on: 020323 Implement-due: 020325 Fix-date: 020325 Fixed-on: (date) In-Test-on: 020325 Test-name: (Numeric id)* Test-system: (hardware) Verify-date: Merge-build-id: Deferred-on: Build-name: (Alpha-numeric identifier) Patch-rel-name: Patch-rel-date: (date) Automated history entries: (Alpha-numeric) Costing Data: Cost: (high, medium, low) Est-fix-hours: Est-fix-date: (date) Est-Num-SLOC: Rev-fix-time: Rev-fix-date: (date) SLOC-Type: SLOC-count: Fix-hours: Miscellaneous Data: Operating-system: Priority: (High, Medium, Low) Enhancement: (Y or N) Workaround: Y Iteration: (Version bug identified in) * May need sanitized 27

Project Specific - Universal IV&V Facility Headline: (text – short problem description) Problem: (text – expanded problem description) Analysis: (text) Resolution: (text) Closure: (text) Submitter-id: Assigned-to: Closer-id: Question: Can Project Specific data be sanitized? 28

Project Specific - Universal IV&V Facility Headline: (text – short problem description) Problem: (text – expanded problem description) Analysis: (text) Resolution: (text) Closure: (text) Submitter-id: Assigned-to: Closer-id: Question: Can Project Specific data be sanitized? 29