Validating and Improving Test-Case Effectiveness Yuri Chernak

Validating and Improving Test-Case Effectiveness Yuri Chernak Presented by Michelle Straughan

Overview Introduction Test-Case Effectiveness Metric (TCE) Improving Test-Case Effectiveness 5 steps to improving TCE Summary

Introduction Software testing is crucial to the success of product Evaluation certifies product usage Verification of test specifications Validation of test cases New metric for evaluating test case Framework for test-case effectiveness

Improving Test-Case Effectiveness – Test-Case Effectiveness Metric Formulation of metric to measure the effectiveness of testing Metric Definition: TCE Ntc /Ntot * 100% TCE value compared to a baseline to determine effectiveness (eg 75% baseline)

Improving Test-Case Effectiveness Analyze causes and areas improvements are needed Proposed improvement framework based on test-escapes (defects test case suite did not find but were side effects found in the same test cycle) Approach relies on analysis of missed defects by test cases

5 steps to Improvement 1. 2. 3. 4. 5. Thoroughly understanding and properly documenting the test process Make Assumptions Gathering defect data and performing causal analysis Identifying main factors Implementing corrective actions

Step 1: Thoroughly understanding and properly documenting the test process Stages in Test Process Description Deliverables Test Planning defines scope, objective and test approach Test Plan Test Design designs test cases Test-Case Specifications Test preparation and execution- preparation and setup of test environment, execution of test cases, finding and reporting defects Defect Report Test evaluation and improvement analyzing testing results Summary Report

Step 2: Make Assumptions After completing step 1, project team makes assumptions about factors affecting TCE Test Planning Test objectives defined in test plan should be traceable to functional specifications Incomplete specifications lead to an incomplete test plan and leads to reduced TCE

Step 2: Make Assumptions Test Design Necessary test cases to test logic of functional features need to be defined, if not adequate coverage will be lacking Test case specifications should correspond to business rules Deficiencies could be caused by: Incomplete test suite (all possible test cases are not defined) Incomplete test cases – proper logic is not tested – negative test cases Incorrect test-specifications Test Preparation and Execution Test cases not executed Test cases improperly executed

Factors affecting test-case effectiveness Figure 1: Factors affecting test-case effectiveness

Step 3: Gathering defect data and performing causal analysis Figure 2: Factors based on causal analysis logic

Step 3: Gathering defect data and performing causal analysis Testers need to track defects and categorize defects. Test-escapes are further categorized according to (fig. 2) Understand reasons for missed defects Verify completeness of test specifications Verify coverage of test-cases Check for test deficiencies Verify the correctness of test cases

Step 4: Identifying main factors Identifying factors that are causing major defects Analyze these defects and focus on these as most needing corrective measures

Step 4: Identifying main factors Figure 3: Pareto chart

Step 5: Implementing corrective actions Depending on project at hand, corrective measures could vary. These may include: Revisiting and revising functional specifications, which results in revising test case specifications Use of traceability matrix to meet coverage needs of the project – ensures that test cases meet business rule Training for testers on proper design techniques, using checklists to help better design test case specifications (involves revisiting and revising test case specifications) Training for testers on proper way to execute tests and verifying test results

Summary New framework is to aid in better effectiveness of test-cases, however its not applicable to some projects as it requires large number of defects Framework would theoretically help to reduce defects present in a product when released to production, since it first checks the test process and if its not effective then provides steps for achieving effectiveness Formalized way of achieving effectiveness, other ways for improving effectiveness could include the use of other testing techniques