@ GMU Automatic Test Generation for Web Applications Jeff Offutt

@ GMU Automatic Test Generation for Web Applications Jeff Offutt Professor, Software Engineering George Mason University Fairfax, VA USA www.cs.gmu.edu/ offutt/ [email protected] Joint research with Blaine Donley, Xiaochen Du, Hong Huang, Upsorn Praphamontripong, Ye Wu, Wuzhi Xu

@ GMU Who Am I PhD Georgia Institute of Technology, 1988 Professor at George Mason University since 1992 – Fairfax Virginia (near Washington DC) Lead the Software Engineering MS & PhD programs – Oldest and largest in USA Editor-in-Chief of Wiley’s journal of Software Testing, Verification and Reliability (STVR) Co-Founder of IEEE International Conference on Software Testing, Verification and Validation (ICST) Co-Author of Introduction to Software Testing (Cambridge University Press) Published over 120 refereed research papers Softec, July 2010 Jeff Offutt 2

@ GMU Test Design in Context Test Design is the process of designing input values that will effectively test software Test design is one of several activities for testing software – Most mathematical – Most technically challenging These ideas are based on my text book with Ammann, Introduction to Software Testing Softec, July 2010 http://www.cs.gmu.edu/ offutt/softwaretest/ Jeff Offutt 3

@ GMU OUTLINE 1.1. How Howisisweb websoftware softwaredifferent? different? 2.2. What Whatresearch researchchallenges challengesdoes doesthe theweb weboffer? offer? 3.3. Bypass Bypasstesting testing(from (fromthe theclient) client) 4.4. Atomic Atomicsection sectionmodeling modeling(on (onthe theserver) server) 5.5. Summary Summary Softec, July 2010 Jeff Offutt 4

@ GMU General Web Terminology Web Page : Data that fits in one browser screen – Static : HTML exists as a file on a computer – Dynamic : Created as needed Web Site : A collection of connected web pages Web Application : A program that is deployed on the web – UI is in HTML – User interacts through HTTP’s request / response cycle Client network Browser javascript Softec, July 2010 Web Server middleware J2EE middleware App Server .Net Java PHP C# Jeff Offutt DB 5

@ GMU Software Deployment Methods Bundled : On your computer when you buy it Shrink-wrapped : Bought at a store on a CD – Downloaded from company’s website or OSS site Contract : Single customer Embedded : Installed on an electronic device Web application : On the web through a URL – Component-based – Concurrent / distributed – One copy on the server – Can be updated at any time (fast update cycle) – User interactive Softec, July 2010 Jeff Offutt 6

Execution Overview @ GMU Incoming request on port 8080 2 3 8 1 Server HTTP Request Request / Response Objects HTTP Response Web server Container engine Create thread / call method 4 Softec, July 2010 Response back to requestor (client) 7 Modified Response Objects Return 6 5 Program component Jeff Offutt 7

@ GMU Web Software Container Engine Container Engine Web App 2 Web App 1 C1a C2a C1b C1c C2c Shared memory C2b C2d Shared memory Shared memory Softec, July 2010 Jeff Offutt 8

@ GMU Three Issues with Programming Web Apps 1. Control flow 2. State management and variable scope 3. Concurrency Softec, July 2010 Jeff Offutt 9

@ GMU Traditional Control Flow Procedural languages – Method / function calls – Decisions – if, while, for, repeat-until, switch, – Static includes – other code pulled in before compiling OO languages – Dynamic binding via polymorphism Client / Server – Message passing Softec, July 2010 Jeff Offutt 10

@ GMU Web App Control Flow (1) Traditional Control Flow Mechanisms 1. Same as traditional – Software on server and client 2. Synchronous message passing – Client to server, HTTP – Also server to other servers 3. Event handling – On the client Softec, July 2010 Jeff Offutt 11

@ GMU Web App Control Flow (2) New Control Flow Mechanisms 4. Asynchronous message passing – Client to server, Ajax 5. Forward – Transfers control from one server component to another, no return 6. Redirect – Ask client to send request elsewhere 7. URL rewriting by users 8. Dynamic include – Control passes to another component, then returns, no parameters 9. Dynamic binding – Reflection allows new components to be added and used dynamically Softec, July 2010 Jeff Offutt 12

@ GMU Ramifications The traditional control flow graph does not model essential parts of web app execution ! UML diagrams do not model many of these Most developers learn the syntax, but not the concepts behind these new control connections Lots of poorly designed software and lots and lots of poorly understood software faults ! Softec, July 2010 Jeff Offutt 13

@ GMU Three Issues with Programming Web Apps 1. Control flow 2. State management and variable scope 3. Concurrency Softec, July 2010 Jeff Offutt 14

@ GMU State & Session Tracking A series of related interactions between a Session : client and a web server Session tracking refers to keeping data between multiple HTTP requests This problem is essential to maintaining state, which we understand quite well in the context of traditional procedural programming and objectoriented programming The Web brings in unique constraints HTTP is stateless Softec, July 2010 Distributed Jeff Offutt 15

@ GMU State in Procedural Languages The C programming language has simple ways to handle state Global variable char name [25]; main () { int x, y, z; Local variables . : We added several layers of scope in OO languages Softec, July 2010 Jeff Offutt 16

@ GMU State in OO Languages (Java) Class 1 Package public members protected members Class 3 default private members inheritance Class 2 Class 5 Softec, July 2010 Class 4 Jeff Offutt 17

@ GMU State in Web Applications These schemes make two simple, subtle, assumptions : 1. The software components share physical memory 2. The program runs to completion with active memory But these assumptions are violated in web applications ! 1. Distributed software components 2. Stateless nature of HTTP To keep state in web applications, we need different ways to store and access variables and objects Public access and parameter passing are not possible in Web applications! Softec, July 2010 Jeff Offutt 18

Sessions—Big Picture @ GMU Time Web Server Client 1 Client 2 Time HTTP Request HTTP Request HTTP Response Session ID 0347 HTTP Response Session ID 4403 Server returns a new HTTP Request unique session IDSession ID 0347 when the request has HTTP Response none HTTP Request Session ID 4403 HTTP Response HTTP Request Session ID 0347 HTTP Request Session ID 4403 HTTP Response HTTP Response Softec, July 2010 Jeff Offutt 19

Sessions—Big Picture @ GMU Time Web Server Client 1 Client 2 Time HTTP Request HTTP Request HTTP Response Session ID 0347 HTTP Response Session ID 4403 HTTP Request HTTP Request Session ID 0347 Client stores the IDHTTP Response and sends it to the HTTP Request server in subsequent Session ID 0347 requests Server recognizes all HTTP Response the requests as being from the same client. This defines a session. Softec, July 2010 Session ID 4403 HTTP Response HTTP Request Session ID 4403 Server recognizes these requests as being from a different HTTP Response client. Jeff Offutt 20

Example @ GMU A small J2EE Web app with 5 software components Servlet S1 Servlet S2 Client JSP 1 JSP 2 JSP 3 How can the software components share data? Softec, July 2010 Jeff Offutt 21

@ GMU Sharing Data : Session Object One program component can store a value in the session object Another component can retrieve, use, and modify the value Depends on the container engine: – Software components run as threads, not processes – Container engine stays resident and can keep shared memory Softec, July 2010 Jeff Offutt 22

@ GMU Sharing Data : Session Object Software components share “container” access data Container engine Servlet 1 session object Servlet 2 Client JSP 1 JSP 2 JSP 3 Softec, July 2010 Jeff Offutt 23

Context Scope @ GMU Container Engine session Servlet S2 object 1 Servlet S1 session object 2 JSP 3 JSP 1 context JSP 2 object Session 1 Softec, July 2010 Context (application) Jeff Offutt Session 2 24

@ GMU JSP Scope & State Management JSPs formalize this with four separate scopes 1. Page : Within the same program component (web page) 2. Request : Within the same request 3. Session : Within all requests from the same session 4. Application : Within all sessions for one application context Each can be used by different sets of program components Some exist for different periods of time Softec, July 2010 Jeff Offutt 25

@ GMU Sharing Data with Scope (JSP) request page forward st e u q re requ est request Client 1 page forward session request page Client 2 Softec, July 2010 application Jeff Offutt 26

@ GMU Control Flow and State Summary Managing state and control flow is fundamental to any program These are the most unique aspects of designing and programming web applications Software vendors are creating new frameworks all the time – Most introduce additional state handling techniques Many professional web developers make fundamental mistakes with state and control ! State management is the most common source of software faults in web applications Softec, July 2010 Jeff Offutt 27

@ GMU Three Issues with Programming Web Apps 1. Control flow 2. State management and variable scope 3. Concurrency Softec, July 2010 Jeff Offutt 28

@ GMU Concurrency in Web Apps Multiple users often access the same web app at the same time Software component objects are run as threads Threads on the same object share some variables This can cause problems when storing data into permanent storage 1. 2. 3. 4. 5. 6. Servlet thread for user 1 reads a data file into memory Servlet thread for user 2 reads the same data file Thread 1 changes the data Thread 2 changes the data Thread 1 writes the file onto disk Thread 2 writes the file onto disk 7. Changes from user 1 are lost ! Programmers are responsible for avoiding this problem Softec, July 2010 Jeff Offutt 29

@ GMU Concurrency Example public class countHellos extends HttpServlet { // Number of times the servlet has been executed since // the servlet object was created private int visit 0; // shared among threads public void doGet (HttpServletRequest req, HttpServletResponse res) throws servletException, IOException { res.setContentType (“text/html; charset \”UTF-8\””); PrintWriter out res.getWriter (); visit ; out.println (“ HTML ”); . Softec, July 2010 Jeff Offutt 30

Concurrency Illustration @ GMU Time Web Server Servlet thread for User 1 Start Servlet thread for User 2 visit 17; visit ; (18) Suspend Start To client Complete visit 18; visit ; (19) Resume To client visit 19; oops !! Complete Softec, July 2010 Jeff Offutt 31

@ GMU Concurrency and (Java) Threads Container engines create separate threads for each HTTP request Some state is unique to each thread (not shared) – Next statement to execute – The call stack : Where the current method will return to Where the calling method returns to Parameter values and local values for each method Some state is shared among threads – Instance variable values (declared outside methods) – Class variables values (declared static outside methods) – Contents of files and other external resources Softec, July 2010 Jeff Offutt 32

@ GMU Concurrency and Synchronization The Java “synchronized” command is used to avoid concurrency problems If a method is synchronized, only one thread can execute the method at a time – Other threads are blocked until the first thread finishes synchronized public void doGet ( . . . Softec, July 2010 Jeff Offutt 33

Concurrency Illustration @ GMU Time Web Server Servlet thread for User 1 Start Servlet thread for User 2 visit 17; visit ; (18) Suspend Start BLOCKED Resume To client visit 18; Complete Resume To client Complete Softec, July 2010 Jeff Offutt yay !! visit 18; visit ; (19) yay !! 34

@ GMU Web Concurrency Summary Most professional programmers have very little knowledge of concurrency Most professional programmers have very little experience with concurrency For many, writing a web app is the first time they use concurrency – And they often don’t realize the potential concurrency! Concurrency is a major source of software faults in web applications Softec, July 2010 Jeff Offutt 35

@ GMU OUTLINE 1.1. How Howisisweb websoftware softwaredifferent? different? 2.2. What Whatresearch researchchallenges challengesdoes doesthe theweb weboffer? offer? 3.3. Bypass Bypasstesting testing(from (fromthe theclient) client) 4.4. Atomic Atomicsection sectionmodeling modeling(on (onthe theserver) server) 5.5. Summary Summary Softec, July 2010 Jeff Offutt 36

@ GMU Testing Web Applications Web applications are heterogeneous, dynamic and must satisfy very high quality attributes Use of the Web is hindered by low quality Web sites and applications Web apps need to be built better and tested more Web site software is extremely loosely coupled – Coupled through the Internet – separated by space – Coupled to diverse hardware and software applications Web software services offer dynamically changing flow of control – Web pages are created by software on user request – Finding all screens in a web app is an undecidable problem Softec, July 2010 Jeff Offutt 37

@ GMU Separation of Concerns Presentation layer HTML, output and UI Data content layer Data representation layer In-memory data storage Data storage layer Permanent data storage Softec, July 2010 Computation, data access Jeff Offutt 38

@ GMU Differences in Testing Web Apps Traditional graphs are not useful – Control flow graph – Call graph State behavior is hard to model and describe All inputs go through the HTML UI – low controllability Hard to get access to server-side state (memory, files, database) – low observability Not clear what logic predicates are useful No model for mutation operators on web software Softec, July 2010 Jeff Offutt 39

@ GMU New Essential Problems 1. Web site software is extremely loosely coupled – Coupled through the Internet – separated by space – Coupled to diverse hardware and software applications – Web services will dynamically couple with other services after deployment – without human intervention ! 2. Web software services offer dynamically changing flow of control – Web pages are created by software on user request – The interaction points (forms, buttons, etc.) vary depending on state : the user, previous choices, server-side data, even time of day – Finding all screens in a web app is an undecidable problem Softec, July 2010 Jeff Offutt 40

@ GMU Extremely Loose Coupling Tight Coupling : Dependencies among the methods are encoded in their logic – Changes in A may require changing logic in B Loose Coupling : Dependencies among the methods are encoded in the structure and data flows – Changes in A may require changing data uses in B Extremely Loose Coupling (ELC): Dependencies are encoded only in the data contents – Changes in A only affects the contents of B’s data Softec, July 2010 Jeff Offutt 41

@ GMU Web Ramifications of ELC applications encourage ELC – Physical separation of hardware and software makes ELC necessary – XML supports ELC ELC has some non-obvious affects – Software modules can dynamically integrate with others if they use the same data structures – EJBs can be inserted into Web applications, which can immediately start using them Softec, July 2010 Jeff Offutt 42

Dynamic Flow of Control @ GMU WebPics WebPics Selamat petang, Penny Tai! Howdy, Paul Ammann! Search Search Recommended Movies Recommended Movies Z ZZ ZZZ A B Examine queue (Warning: Queue empty) Examine queue View account View account C D Frequent customer bonus How can we ensure the reliability of this type of system? Softec, July 2010 Jeff Offutt 43

@ GMU OUTLINE 1.1. How Howisisweb websoftware softwaredifferent? different? 2.2. What Whatresearch researchchallenges challengesdoes doesthe theweb weboffer? offer? 3.3. Bypass Bypasstesting testing(from (fromthe theclient) client) 4.4. Atomic Atomicsection sectionmodeling modeling(on (onthe theserver) server) 5.5. Summary Summary Softec, July 2010 Jeff Offutt 44

@ GMU Validating Inputs Input Validation Deciding if input values can be processed by the software Before starting to process inputs, wisely written programs check that the inputs are valid – How should a program recognize invalid inputs ? – What should a program do with invalid inputs ? Web apps are user input-driven, so input validation is critical to their success If the input space is described as a grammar, a parser can check for validity automatically – This is very rare – It is easy to write input checkers –also easy to make mistakes Softec, July 2010 Jeff Offutt 45

@ GMU Representing Input Domains Desired inputs (goal domain) Described inputs (specified domain) Accepted inputs (implemented domain) Softec, July 2010 Jeff Offutt 46

Representing Input Domains @ GMU Goal domains are often irregular Goal domain for credit cards† – First digit is the Major Industry Identifier – First 6 digits and length specify the issuer – Final digit is a “check digit” – Other digits identify a specific account Common specified domain – First digit is in { 3, 4, 5, 6 } (travel and banking) – Length is between 13 and 16 Common implemented domain – All digits are numeric All digits are numeric † Softec, July 2010 More details are on : http://www.merriampark.com/anatomycc.htm Jeff Offutt 47

@ GMU Representing Input Domains goal domain specified domain This region is a rich source of software errors implemented domain Softec, July 2010 Jeff Offutt 48

@ GMU How to Apply to Web Apps? Web applications encode many of the rules that define the valid input space in the UI – The UI is implemented in HTML and available in the user’s browser Users can : – “view HTML” – Save HTML – Modify HTML – Reload it and run the program with a modified UI Softec, July 2010 Jeff Offutt 49

@ GMU FORM Abbreviated HTML INPUT Type “text” Name “username” Size 20 INPUT Type “text” Name “age” Size 3 Maxlength 3 P Version to purchase: INPUT Type “radio” Name “version” Value “150” Checked INPUT Type “radio” Name “version” Value “250” INPUT Type “radio” Name “version” Value “500” INPUT Type “submit” onClick “return checkInfo(this.form)” INPUT Type “hidden” isLoggedIn “no” /FORM Softec, July 2010 Jeff Offutt 50

@ GMU FORM Saved & Modified HTML INPUT Type “text” Name “username” Size 20 INPUT Type “text” Name “age” Size 3 Maxlength 3 P Version to purchase: Allows an input with arbitrary age, no checking, cost 25 ‘ ‘ can crash an XML parser INPUT Type “radio” Name “version” Value “150” Text fields can have SQL statements INPUT Type “radio” Name “version” Value “250” INPUT Type “radio” Name “version” Value “500” Checked 25 INPUT Type “submit” onClick “return checkInfo (this.form)” INPUT Type “hidden” isLoggedIn “no” yes /FORM Softec, July 2010 Jeff Offutt 51

SQL Injection @ GMU User Name: turing ’ OR ‘1’ ‘1 Password: enigma ’ OR ‘1’ ‘1 Original SQL: SELECT username FROM adminuser WHERE username ‘turing’ AND password ‘enigma’ “injected” SQL: SELECT username FROM adminuser WHERE username ‘turing’ OR‘1’‘1’ ‘1’‘1’ AND password ‘enigma’ OR OR OR‘1’ ‘1’ ‘1’ ‘1’ Softec, July 2010 Jeff Offutt 52

@ GMU Bypass Testing This example illustrates how users can “bypass” client-side constraint enforcement Bypass testing constructs tests to intentionally violate constraints – Eases test automation – Checks robustness – Evaluates security Preliminary results – Rules for constructing tests – Successfully found errors in numerous Web apps Softec, July 2010 Jeff Offutt 53

@ GMU Applying Bypass Testing Validating input data on the client is like asking your opponent to hold your shield in a sword fight Bypass testing designs tests to violate constraints – Eases test automation – Checks robustness – Evaluates security Analyze HTML to extract each form element Model constraints imposed by HTML and JavaScript Rules for data generation : – From client-side constraints – Typical security violations – Common input mistakes Softec, July 2010 Jeff Offutt 54

@ GMU Real-World Examples atutor.ca Atalker demo.joomla.or Poll, Users phpMyAdmin Main page, Set Theme, SQL Query, DB Stats brainbench.com Submit Request Info, New user myspace.com Events & Music Search Softec, July 2010 nytimes.com Us-markets mutex.gmu.edu Login form yahoo.com Notepad, Composer, Search reminder, Weather Search barnesandnoble.com Cart manager, Book search/results bankofamerica.com ATM locator, Site search comcast.com Service availability ecost.com Detail submit, Shopping cart control google.com Froogle, Language tools Pure black-box testingpageflakes.com Registration amazon.com means wellsfargolife.com Item dispatch, no source (or buy permission) needed ! search Quote Handle Jeff Offutt 55

@ GMU Output Checking (V) Valid Responses : invalid inputs are adequately processed by the server (V1) Server acknowledges the invalid request and provides an explicit message regarding the violation (V2) Server produces a generic error message (V3) Server apparently ignores the invalid request and produces an appropriate response (V4) Server apparently ignores the request completely (F) Faults & Failures : invalid inputs that cause abnormal server behavior (typically caught by web server when application fails to handle the error) (E) Exposure : invalid input is not recognized by the server and abnormal software behavior is exposed to the users These do not capture whether the valid responses corrupted data on the server Softec, July 2010 Jeff Offutt 56

@ GMU Results v Softec, July 2010 Jeff Offutt 57

@ GMU OUTLINE 1.1. How Howisisweb websoftware softwaredifferent? different? 2.2. What Whatresearch researchchallenges challengesdoes doesthe theweb weboffer? offer? 3.3. Bypass Bypasstesting testing(from (fromthe theclient) client) 4.4. Atomic Atomicsection sectionmodeling modeling(on (onthe theserver) server) 5.5. Summary Summary Softec, July 2010 Jeff Offutt 58

@ GMU Dynamic Execution of Web Apps Parts of the program are generated dynamically Dynamic web pages are created from user requests Different users will see different programs ! Users can make unexpected changes to the flow of control – Operational transitions : Transitions NOT based on an HTML link: back-button, URL rewriting, refresh Potential flow of control cannot be known statically Softec, July 2010 Jeff Offutt 59

@ GMU Control Flow Graphs in Web Applications Many testing criteria on non-Web software rely on a static control flow graph – Edge testing, data flow, logic coverage – Also slicing, change impact analysis, Static control flow graphs cannot be computed for Web applications! But all the pieces of the web pages and the programs are contained in the software Potential flow of control cannot be known statically Softec, July 2010 Jeff Offutt 60

Atomic Sections @ GMU PrintWriter out response.getWriter(); P1 out.println (“ HTML ”) out.println (“ HEAD TITLE ” title title “ /TITLE /HEAD ”) out.println (“ BODY ”) if (isUser) { Atomic sections P2 out.println (“ CENTER Welcome! /CENTER ”); for (int i 0; i myVector.size(); i ) if (myVector.elementAt(i).size 10) P3 out.println (“ p b ” myVector.elementAt(i) myVector.elementAt (i)“ /b /p ”); else P4 Empty atomic section out.println (“ p " myVector.elementAt (i) (i)“ /p ”); myVector.elementAt } else P5 P6 { } out.println (“ /BODY /HTML ”); Content variables out.close (); Softec, July 2010 Jeff Offutt 61

@ GMU Atomic Sections Defined A section of HTML with the property that if any part of the section is sent to a client, the entire section is – May include JavaScript – All or nothing property An HTML file is an atomic section Content variable : A program variable that provides data to an atomic section Atomic sections may be empty Softec, July 2010 Jeff Offutt 62

Composite Sections @ GMU Atomic sections are combined to represent all possible screens the program can generate Much like a CFG represents all paths through a unit – Four ways to combine: 1. 2. 3. 4. Sequence : p1 p2 Selection : (p1 p2) Iteration : p1* Aggregation : p1 {p2} – p2 is included inside of p1 The previous example produces: p p1 (p2 (p3 p4)* p5) p6 Composite sections can be generated automatically Softec, July 2010 Jeff Offutt 63

Modeling Dynamic Interaction Three types of transitions @ GMU 1. Link Transition : An HTML link 2. Composite Transition : Execution of a software component causes a composite section to be sent to the client 3. Operational Transition : A transition out of the software’s control Softec, July 2010 Back button Refresh button User edits the URL (URL rewriting) Browser reloads from cache Jeff Offutt 64

@ GMU Inter-Component Modeling A graphical representation of the entire web application A Web Application Graph (WAG) – Nodes are web components – Edges are transitions Three types of transitions 1. Static links 2. Dynamic links 3. Forwarding links Information on links – Type of HTTP request – Data being transmitted as parameters Current State : Static variables and session information Softec, July 2010 Jeff Offutt 65

@ GMU Tests level Test Criteria can be applied at intra- and inter-component Tests are created by deriving sequences of transitions among the Web software components and composite sections Softec, July 2010 Jeff Offutt 66

@ GMU WAG (Inter-Component) Tests L1 : Evaluate static link transitions – One test generated for each form – Typical tests from previous test criteria L2 : L1 with two extensions – Values entered with URL rewriting – Multiple tests for each form L3 : Operational transitions – Starting on non-initial pages, no subsequent transitions L4 : Operational transitions – L1 tests with one operational transition at end L5 : L4 tests to traverse every transition out of the final page Softec, July 2010 Jeff Offutt 67

@ GMU Empirical Evaluation Testing STIS STIS helps users keep track of arbitrary textual information 18 JSPs, 5 Java classes, database Atomic sections derived automatically – Parser works on Java servlets, JSPs, Java classes WAG derived by hand Form data chosen by hand 109 total tests Softec, July 2010 Jeff Offutt 68

@ GMU STIS Web Application Graph index.jsp post (userid, password) record add.jsp post (name, category, content) record insert.jsp static link transition Softec, July 2010 login.jsp logout.jsp browse.jsp categories.jsp post (category, search name) update search params.jsp forward link transition Jeff Offutt post (action, categoryName) dynamic link transition 69

@ GMU Results from Testing STIS previous web tests Failure Category Number of tests 1. Pages displayed without authentication 2. Records added without authentication 3. Runtime failures (unhandled exceptions) Total number of failures 109 tests L1 L2 L3 L4 L5 29 21 7 19 33 0 0 2 4 4 0 0 1 2 0 0 3 2 5 2 0 3 5 11 6 Found 25 naturally occurring failures Softec, July 2010 Jeff Offutt 70

Atomic Sections Summary @ GMU Atomic sections fundamentally model Web applications – Allow the Web app form of CFGs Can also be used for – Software evolution – Design modeling / evaluation – Change impact analysis (slicing) – Coupling of Web application components Softec, July 2010 Jeff Offutt 71

@ GMU OUTLINE 1.1. How Howisisweb websoftware softwaredifferent? different? 2.2. What Whatresearch researchchallenges challengesdoes doesthe theweb weboffer? offer? 3.3. Bypass Bypasstesting testing(from (fromthe theclient) client) 4.4. Atomic Atomicsection sectionmodeling modeling(on (onthe theserver) server) 5.5. Summary Summary Softec, July 2010 Jeff Offutt 72

Cost of Late Testing @ GMU 60 Assume 1000 unit cost, per fault, 100 faults 3 6 50 40 30 6K 1 3K 2 0K 10 0 25 K 0K 0K Fault Origin (%) Fault Detection (%) Unit Cost (X) 20 10 0 Reqs Design Prog / Unit Integration Testing Testing System Testing Production Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002 Softec, July 2010 Jeff Offutt 73

@ GMU Advantages of Criteria-Based Test Design Criteria maximize the “bang for the buck” – Fewer tests that are more effective at finding faults Comprehensive test set with minimal overlap Traceability from software artifacts to tests – The “why” for each test is answered – Built-in support for regression testing A “stopping rule” for testing—advance knowledge of how many tests are needed Natural to automate Softec, July 2010 Jeff Offutt 74

@ GMU Criteria Summary Many Manycompanies companiesstill stilluse use“monkey “monkeytesting” testing” AAhuman humansits sitsatatthe thekeyboard, keyboard,wiggles wigglesthe themouse mouseand andbangs bangsthe the keyboard keyboard No Noautomation automation Minimal Minimaltraining trainingrequired required Some Somecompanies companiesautomate automatehuman-designed human-designedtests tests But Butcompanies companiesthat thatalso alsouse useautomation automationand andcriteria-based criteria-basedtesting testing Save money Find more faults Build better software Softec, July 2010 Jeff Offutt 75

Open Questions @ GMU How to define data flow ? – DU-pairs cannot be determined statically – uses cannot always be found Testing research issues not addressed yet : – Session data – Multiple users – Concurrency – Input data – Output validation – AJax Softec, July 2010 Jeff Offutt 76

@ GMU Conclusions The Web provides a new way to deploy software The new technologies mean that old testing techniques do not work very well New tools and techniques are being developed Most are still in the research stage Most companies test web software very badly Softec, July 2010 Jeff Offutt 77

@ GMU References and Contact Modeling Presentation Layers of Web Applications for Testing, Jeff Offutt and Ye Wu, Springer’s Software and Systems Modeling, 9(2), April 2010 Testing Web Services by XML Perturbation, Wuzhi Xu, Jeff Offutt and Juan Luo, ISSRE 2010 Testing Web Applications by Modeling with FSMs, Anneliese Andrews, Jeff Offutt and Roger Alexander, Springer’s Software Systems and Modeling, 4(3):326-345, July 2005. Bypass Testing of Web Applications, Jeff Offutt, Ye Wu, Xiaochen Du and Hong Huang, ISSRE 2004 Generating Test Cases for Web Services Using Data Perturbation, Jeff Offutt and Wuzhi Xu, TAV-Web, July 2004 Quality Attributes of Web Software Applications, Jeff Offutt, IEEE Software: Special Issue on Software Engineering of Internet Software, March/April 2002 Generating Test Cases for XML-based Web Component Interactions Using Mutation Analysis, Suet Chun Lee and Jeff Offutt, ISSRE 2001 Jeff Offutt [email protected] http://cs.gmu.edu/ offutt/ Softec, July 2010 Jeff Offutt 78