Black-box Testing for Evolving COTS-Based Software Ye Wu

Black-box Testing for Evolving COTS-Based Software Ye Wu Information and Software Engineering Department George Mason University

Agenda Introduction Challenges Methodology Conclusion IWICSS Dr. Ye Wu 2

What is COTS? COTS – Commercial-Off-The-Shelf COTS describes ready-made products that can easily be obtained. IWICSS Dr. Ye Wu 3

Main characteristics It exists a priori Used without source code modification It is available to the general public It can be bought (or leased or licensed). Supported and evolved by vendor IWICSS Dr. Ye Wu 4

Challenges Facing Evolving COTS-based Software What should COTS-users do if COTS products are changed? COTS-user transparent changes Changes that affect COTS product users, but do not affect user system architecture Changes that affects COTS user architecture IWICSS Dr. Ye Wu 5

COTS-user transparent changes Mainly corrective maintenance and a small part of the changes in perfective and adaptive maintenance activities No change is required for COTS users IWICSS Dr. Ye Wu 6

COTS-user aware changes Poor design in the previous version Technology changes Changes of standards Sometimes a strategic move to lock in customers IWICSS Dr. Ye Wu 7

COTS-user aware changes Changes that affect COTS users, but do not affect COTS users’ architecture Changes that affect COTS users’ architecture IWICSS Dr. Ye Wu 8

Black-box Testing Strategies Random Testing Partition Testing Boundary value Testing Decision Table Testing IWICSS Dr. Ye Wu 9

Black-box Testing for Evolving COTS-based Software Information about COTS products to perform adequate black-box testing is not always available Customization needs to be done on some COTS products IWICSS Dr. Ye Wu 10

COTS Product Modifications Generalization Modifications Specialization Modifications Reconstruction Modifications IWICSS Dr. Ye Wu 11

Black-box Testing Methods for Components Partition Testing of Generalization Modification: P1 P2 p1 p3 P3 Before generalization modification IWICSS P1 P’ P2 P1 P’ p2 P’ P2 p4 P4 P3 Case (i): After generalization modification P4 P3 Case (ii): After generalization modification Dr. Ye Wu P4 P3 P4 Case (iii): After generalization modification 12

Black-box Testing Methods for Components Partition Testing for Specialization modification: P1 P2 p1 p2 p3 p4 P3 P’ P2 P1 P’ P2 P4 Before specialization modification IWICSS P1 P3 P4 Case (i): After specialization modification Dr. Ye Wu P3 P4 Case (ii): After specialization modification 13

Black-box Testing Methods for Components Partition Testing for Reconstruction modification: P1 p1 p2 p3 p4 P3 P2 P1 P2 P1 P2 P4 P3 P4 P3 P4 Before reconstruction modification IWICSS Special case: After reconstruction modification Dr. Ye Wu General case: After reconstruction modification 14

Initial Results Total 48 partitions – old version 17 changed or added partitions – new version 34 faulty versions (each with one inserted fault) 21 faults identified 13 faults unidentified 5 faults can be detected during Unittesting IWICSS Dr. Ye Wu 15

Testing Framework for Evolving COTS-based Software Normal Interfaces Client Testing Interfaces Test Case Repository Component COTS Developer Delivered Delivered Version Version Test Case Analyzer Test Case Test Analyzer Specification Old Version new Version IWICSS Dr. Ye Wu 16

Conclusion and Future Work Adopting evolving COTS products can be risky if without adequate testing Black-box testing can be adopted, but need to be adequately adjusted Automated test framework is required IWICSS Dr. Ye Wu 17