Software Testing JUnit Ioanna Miliou Giuseppe Attardi Advanced

Software Testing JUnit Ioanna Miliou Giuseppe Attardi Advanced Programming Università di Pisa

Testing Testing is the activity of finding out whether a piece of code (a method, a class, or program) produces the intended behavior.

Edsger Dijkstra

Economics The earlier a defect (bug) is found, the cheaper it is to fix it! Cost to fix a defect Time detected Time introduced Requirements Architecture Construction Requirements 1x - - Architecture 3x 1x - Construction 5-10x 10x 1x System test 10x 15x 10x Post-release 10-100x 25-100x 10-25x A study conducted by NIST in 2002 reports that software bugs cost the U.S. economy 59.5 billion annually. Source: McConnell, Steve (2004). Code Complete (2nd ed.). Microsoft Press.

Outline Testing overview Test methods Black-box vs White-box testing Test levels Unit vs Integration vs System Testing Test types Automated testing JUnit overview What, Why, Who? Classes Tests Fixtures Test Suites Examples Demo

Test Methods - The box approach

Black-box testing Treats software under test as a black-box without knowing its internals. Tests are using software interfaces and trying to ensure that they work as expected. As long as functionality of interfaces remains unchanged, tests should pass even if internals are changed. Tester is aware of what the program should do but does not have the knowledge of how it does it. It provides external perspective of the software under test.

Black-box testing Some of the advantages of black-box testing are: Efficient for large segments of code Code access is not required Separation between user’s and developer’s perspectives Some of the disadvantages of black-box testing are: Limited coverage since only a fraction of test scenarios is performed Inefficient testing due to tester’s lack of knowledge about software internals Blind coverage since tester has limited knowledge about the application

White-box testing It looks inside the software and uses that knowledge as part of the testing process. If, for example, exception is thrown under certain conditions, the test might want to reproduce those conditions. Requires internal knowledge of the system and programming skills. It provides internal perspective of the software under test.

White-box testing Some of the advantages of white-box testing are: Efficient in finding errors and problems Required knowledge of internals of the software under test is beneficial for thorough testing Allows finding hidden errors Programmers introspection Helps optimizing the code Due to required internal knowledge of the software, maximum coverage is obtained Some of the disadvantages of white-box testing are: Might not find unimplemented or missing features Requires high level knowledge of internals of the software under test Requires code access

Test Levels Unit testing on individual units of source code ( smallest testable part) Integration testing on groups of individual software modules System testing on a complete, integrated system (evaluate compliance with requirements)

Unit Testing Tests the smallest individually executable code units. Usually done by programmers. Test cases might be selected based on code, specification, intuition, etc. Tools: – Test driver/harness – Code coverage analyzer – Automatic test case generator

Integration Testing Tests interactions between two or more units or components. Usually done by programmers. Emphasizes interfaces. Issues: – In what order are units combined? – How do you assure the compatibility and correctness of externally-supplied components?

Integration Testing How are the units integrated? What are the implications of this order? Top-down need stubs; top-level tested repeatedly Bottom-up need drivers; bottom-levels tested repeatedly Critical units first stubs and drivers needed; critical units tested repeatedly

Integration Testing Potential problems: Inadequate unit testing Inadequate planning & organization for integration testing Inadequate documentation & testing of externallysupplied components

System Testing Test the functionality of the entire system. Usually done by professional testers. Realities: Not all problems will be found no matter how thorough or systematic the testing. Testing resources (staff, time, tools, labs) are limited. Specifications are frequently unclear/ambiguous and changing (and not necessarily complete and up-to-date). Systems are almost always too large to permit test cases to be selected based on code characteristics. Exhaustive testing is not possible. Testing is creative and difficult. Testing must be planned.

System Testing Types: Alpha Testing It is carried out by the test team within the developing organization. Beta Testing It is performed by a selected group of friendly customers. Acceptance Testing It is performed by the customer to determine whether to accept or reject the delivery of the system. Performance Testing It is carried out to check whether the system meets the nonfunctional requirements identified in the SRS document

Different Types of Testing Installation Testing Smoke and Sanity Testing Functional & Non-Functional Testing Continuous Testing Destructive Testing Usability Testing Development Testing A/B Testing Concurrent Testing Regression Testing

Installation, Smoke & Sanity Testing Installation Testing An installation test assures that the system is installed correctly and working at actual customer's hardware. Smoke Testing Smoke testing consists of minimal attempts to operate the software, designed to determine whether there are any basic problems that will prevent it from working at all. Such tests can be used as build verification test. Sanity Testing Sanity testing determines whether it is reasonable to proceed with further testing.

Functional & Non-Functional Testing Functional testing It refers to activities that verify a specific action or function of the code. These are usually found in the code requirements documentation, although some development methodologies work from use cases or user stories. Functional tests tend to answer the question of "can the user do this" or "does this particular feature work.” Non-functional testing It refers to aspects of the software that may not be related to a specific function or user action, such as scalability or other performance, behavior under certain constraints, or security. Testing will determine the breaking point, the point at which extremes of scalability or performance leads to unstable execution. Nonfunctional requirements tend to be those that reflect the quality of the product, particularly in the context of the suitability perspective of its users.

Continuous, Destructive & Usability Testing Continuous testing It is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. Destructive testing It attempts to cause the software or a sub-system to fail. It verifies that the software functions properly even when it receives invalid or unexpected inputs, thereby establishing the robustness of input validation and error-management routines. Usability testing It is to check if the user interface is easy to use and understand. It is concerned mainly with the use of the application.

Development, A/B & Concurrent Testing Development Testing It is a software development process that involves synchronized application of a broad spectrum of defect prevention and detection strategies in order to reduce software development risks, time, and costs. A/B testing It is basically a comparison of two outputs, generally when only one variable has changed: run a test, change one thing, run the test again, compare the results. This is more useful with more small-scale situations, but very useful in fine-tuning any program. Concurrent testing The focus is on the performance while continuously running with normal input and under normal operational conditions, as opposed to stress testing, or fuzz testing. Memory leak, as well as basic faults are easier to find with this method.

Regression Testing Test modified versions of a previously validated system. Usually done by testers. The goal is to assure that changes to the system have not introduced errors (caused the system to regress). The primary issue is how to choose an effective regression test suite from existing, previously-run test cases.

Test Automation Test execution: Run large numbers of test cases/suites without human intervention. Test generation: Produce test cases by processing the specification, code, or model. Test management: Log test cases & results; map tests to requirements & functionality; track test progress & completeness

Why should tests be automated? More testing can be accomplished in less time. Testing is repetitive, tedious, and error-prone. Test cases are valuable – once they are created, they can and should be used again, particularly during regression testing.

Test Automation Issues Does the payoff from test automation justify the expense and effort of automation? Learning to use an automation tool can be difficult. Tests, have a finite lifetime. Completely automated execution implies putting the system into the proper state, supplying the inputs, running the test case, collecting the results, and verifying the results.

What is JUnit? JUnit is an open source framework that has been designed for the purpose of writing and running tests in the Java programming language. Originally written by Erich Gamma and Kent Beck University of Calgary.

Regression-Testing Framework JUnit is a regression-testing framework that developers can use to write unit tests as they develop systems. This framework creates a relationship between development and testing. You start coding according to the specification and need. After that you use the JUnit test runners to verify how much it deviates from the intended goal. This really helps to develop test suites that can be run any time when you make any changes in your code. This all process will make you sure that the modifications in the code will not break your system without your knowledge.

Why Use JUnit? Graphical user interface (GUI) which makes it write and test source code quickly and easily. possible to Write and run repeatable tests. Programmers normally write their code, then write tests for it. Better to write tests while writing code! JUnit provides a framework to keep tests small and simple to test specific areas of code. JUnit shows test progress in a bar that is green if testing is going fine and it turns red when a test fails. Multiple tests can run concurrently. The simplicity of JUnit makes it possible for the software developer to easily correct bugs as they are found.

GUI

JUnit helps the programmer: Define and execute tests and test suites Formalize requirements and clarify architecture Write and debug code Integrate code and always be ready to release a working version Typically, in a unit testing, we start testing after completing a module but JUnit helps us to code and test both during the development. So it sets more focus on testing the fundamental building blocks of a system i.e. one block at a time rather than module level functional testing.

Download & Installation of JUnit Eclipse has JUnit already included, therefore if you intend to use JUnit within Eclipse you don't have to download anything. http://www.junit.org/junit4/ To download and install JUnit you currently have the following options – Plain-old JAR Download the jar and add junit-4.12.jar to the CLASSPATH. – Maven Add a dependency to junit:junit in test scope. dependency groupId junit /groupId artifactId junit /artifactId version 4.12 /version scope test /scope /dependency – Gradle Gradle Dependency : Make sure these lines are in your build.gradle apply plugin: 'java’ dependencies { testCompile 'junit:junit:4.12’ } Execution : ./gradle test (see Gradle Java tests for configuration information)

Guidelines for using JUnit Make sure to test before and after integration with other modules. Assume a feature DOESN’T work until you have proven that it does by testing everything that could possibly break. Use informative test names. Try to test only one thing per method to make it easier to find where error occurred. Write tests immediately after you write the function, when it’s fresh in your mind.

What JUnit does JUnit runs a suite of tests and reports results. For each test in the test suite: JUnit calls setUp() – This method should create any objects you may need for testing JUnit calls tearDown() – This method should remove any objects you created JUnit calls one test method – The test method may comprise multiple test cases; that is, it may make multiple calls to the method you are testing – In fact, since it’s your code, the test method can do anything you want – The setUp() method ensures you entered the test method with a virgin set of objects; what you do with them is up to you

JUnit framework T e s tin g c lie n t T est T e s tC a s e r u n ( T e s t R e s u lt ) s e tU p ( ) r u n T e s t( ) te a r D o w n ( ) T e s tS u ite T e s t R e s u lt s e tU p ( ) r u n T e s t( ) te a r D o w n ( ) C o n c re te T e s tC a s e T e s te d C la s s a c t io n ( ) s e tU p ( ) r u n T e s t( ) t e a r D o w n( ) te s t1 ( ) te s t2 ( ) fN a m e r u n T e s t( ) fT e s ts te s t1 ( ) or te s t2 ( ) r u n ( T e s t R e s u lt ) a d d T e s t (T e s t ) f o r a ll t e s t in f T e s t s t e s t . r u n ( T e s t R e s u lt )

Creating a test class in JUnit Override the setUp() method to initialize object(s) under test. Override the tearDown() method to release object(s) under test. Define one or more public testX() methods that exercise the object(s) under test and assert expected results. Define a static suite() factory method that creates a TestSuite containing all the testX() methods. Optionally define a main() method that runs the test in batch mode.

Coding Convention for JUnit Class Name of the test class must end with "Test". Name of the method must begin with "test". Return type of a test method must be void. Test method must not throw any exception. Test method must not have any parameter. Test methods must be annotated by the @Test annotation.

Standards for unit tests You should write a test case for the method which is less dependent on sub method. If you write a test case for dependent methods, your test may fail because the sub method can return wrong value. So, it may take longer to find out the source of the problem. Each unit test should be independent of other test. If you write one unit test for multiple methods then the result may be confusing. Test behavior is more important than methods. Each test case has to be in the same package as the code to be tested. Although, it is nice to cover most of the code in the test cases, it is not advisable to use 100% of the code. If the method is too simple then there is no point of writing unit test for it.

Example: Currency Test that the sum of two Moneys with the same currency contains a value which is the sum of the values of the two Moneys. But what if you have two or more tests that operate on the same or similar sets of objects?!

Fixtures A fixture is just the code you want to run before every test. It’s all the things that must be in place in order to run a test and expect a particular outcome. In JUnit a test fixture is a Java object. You get a fixture by overriding the method protected void setUp() { } The general rule for running a test is: protected void runTest() { setUp(); run the test tearDown(); } so we can override setUp and/or tearDown, and that code will be run prior to or after every test case With older versions of JUnit, fixtures had to inherit from junit.framework.TestCase, but the new tests using JUnit 4 should not do this.

Example: Currencies You want to write several test cases that want to work with different combinations of 12 Swiss Francs, 14 Swiss Francs, and 28 US Dollars. First you create a fixture: Once you have the Fixture in place, you can write as many Test Cases as you'd like. Add as many test methods (annotated with @Test) as you'd like.

Implementing setUp() & tearDown() method Override setUp() to initialize the variables, and objects Since setUp() is your code, you can modify it any way you like (such as creating new objects in it) Reduces the duplication of code In most cases, the tearDown() method doesn’t need to do anything. The next time you run setUp(), your objects will be replaced, and the old objects will be available for garbage collection. Like the finally clause in a try-catch-finally statement, tearDown() is where you would release system resources (such as streams)

The framework resides under package junit.framework for JUnit 3.8 and earlier, and under package org.junit for JUnit 4 and later. Test methods must be annotated by the @Test annotation. Methods to execute before (or after) each (or all) of the test methods with the @Before (or @After) and @BeforeClass (or @AfterClass) annotations.

The structure of a test method A test method doesn’t return a result If the tests run correctly, a test method does nothing If a test fails, it throws an AssertionFailedError The JUnit framework catches the error and deals with it, so you don’t have to do anything

Example: Calculator package org.example.antbook.junit; import org.junit.*; public class TestCalculator { public TestCalculator(String name) { super(name); } Calculator calculator; @Override protected void setUp() throws Exception { calculator new Calculator(); } public void testAdd() { // int sum calculator.add(8, 7); // assertEquals(sum, 15); assertEquals(15, calculator.add(8, 7)); } @Override protected void tearDown() throws Exception { // Clean up, like calculator null; } } Override setUp() method to perform initialization Define one or more testX() methods that can perform the tests and assert expected results Override tearDown() method to clean up

Test Suites In practice, you want to run a group of related tests (e.g. all the tests for a class). Test Test Function To do so, group your test methods in a class. The Suite allows to run collections of tests and/or other suites in a hierarchically or thematically structured way. Module Function Module SRunner Module

Writing Tests for JUnit Need to use the methods of the assert class – javadoc gives a complete description of its capabilities Each test method checks a condition (assertion) and reports back to the test runner whether the test failed or succeeded The test runner uses the result to report to the user (in command line mode) or update the display (in an IDE) All of the methods return void A few representative methods of org.junit.assert – assertTrue(boolean) – assertTrue(String, boolean) – assertEquals(Object, Object) – assertNull(Object) – Fail(String) All the above may take an optional String message as the first argument, for example, static void assertTrue(String message, boolean test)

Sample Assertions static void assertEquals (boolean expected, boolean actual) Asserts that two booleans are equal static void assertEquals (byte expected, byte actual) Asserts that two bytes are equal static void assertEquals (char expected, char actual) Asserts that two chars are equal static void assertEquals (double expected, double actual, double delta) Asserts that two doubles are equal concerning a delta static void assertEquals (float expected, float actual, float delta) Asserts that two floats are equal concerning a delta static void assertEquals (int expected, int actual) Asserts that two ints are equal For a complete list, see: – http://junit.sourceforge.net/javadoc/org/junit/Assert.html

Organize The Tests Create test cases in the same package as the code under test. For each Java package in your application, define a TestSuite class that contains all the tests for validating the code in the package. Define similar TestSuite classes that create higher-level and lowerlevel test suites in the other packages (and sub-packages) of the application. Make sure your build process includes the compilation of all tests.

Example: Counter class For the sake of example, we will create trivial “counter” class and test a – The constructor will create a counter and set it to zero – The increment method will add one to the counter and return the new value – The decrement method will subtract one from the counter and return the new value We write the test methods before we write the code – This has the advantages described earlier – Depending on the JUnit tool we use, we may have to create the class first, and we may have to populate it with stubs (methods with empty bodies) Don’t be alarmed if, in this simple example, the JUnit tests are more code than the class itself

The Counter class itself public class Counter { int count 0; public int increment() { return count; } public int decrement() { return --count; } public int getCount() { return count; } }

JUnit tests for Counter

Running Tests How do you run your tests and collect their results? Once you have tests, you'll want to run them. JUnit provides tools to define the suite to be run and to display its results. To run tests and see the results on the console, run this from a Java program: org.junit.runner.JUnitCore.runClasses(TestClass1.class, .); or this from the command line, with both your test class and junit on the classpath: java org.junit.runner.JUnitCore TestClass1 [.other test classes.] You make your JUnit 4 test classes accessible to a TestRunner designed to work with earlier versions of JUnit, declare a static method suite that returns a test. public static junit.framework.Test suite() { return new JUnit4TestAdapter(Example.class); }

Example: Shopping Cart

Part I : Initialize

Test Fixture

Part 2: Write Test Cases fail(msg) : triggers a failure named msg assertTrue(msg, b) : triggers a failure, when condition b is false assertEquals(msg, v1, v2) : triggers a failure, when v1 v2 assertEquals(msg, v1, v2, ε) : triggers a failure, when v1 v2 ε assertNull(msg, object) : triggers a failure, when object is not null assertNonNull(msg, object) : triggers a failure, when object is null The parameter msg is optional.

Test: Add product

Test: Remove product

Test: Exception handling

Part 3: Write a Test Suite The method suite() assembles the test methods into a test suite – using reflection all methods named testX() will be part of the test suite.

Part 4: Execute Test Suite Execute the main() method to run the tests: java ShoppingCartTest

Demo

xUnit for Other Languages Ada (AUnit) C (CUnit) C# (NUnit) C (CPPUnit, CxxTest) Erlang (EUnit) Fortran (fUnit, pFUnit) Delphi (DUnit) Free Pascal (FPCUnit) Haskell (HUnit) JavaScript (JSUnit) Microsoft .NET (NUnit) Objective-C (OCUnit) OCaml (OUnit) Perl (Test::Class and Test::Unit) PHP (PHPUnit) Python (PyUnit) R (RUnit) Ruby (Test::Unit)

Some Important Tips You should use object oriented best practice for your test code. You should bring all of the common functionalities in the base class. That way you can eliminate duplicate code, long methods etc. You should use proper method name. it is going to be easier for the team to maintain the code in future. The JUnit framework automatically handles the exception. It is good idea not to throw an exception in a test that shouldn't throw an exception. Otherwise, it can hide bugs which may be discovered much later stage. JUnit doesn't provide all of the functionalities to test your code. So, you need an additional test tool. Some of the tools are DBunit, Selenium etc that can be used as an extension of JUnit. Instead of rely on System.out.println() method to figure out the success of the test, you can rely upon the 'assert' method of JUnit so that you can run the automated test and that way the tester doesn't need to monitor the test in person all the time.

Best Practices

Summary Testing Unit testing Integration testing System testing JUnit Test cases Test suites Fixtures Best practices Literature @ junit.org JUnit test Infected JUnit A Cook’s Tour Self-study questions How can we test protected and private methods? What distinguishes a good test suite from a bad one?

Thanks Kent Beck, Erich Gamma – JUnit Cookbook http://junit.sourceforge.net/doc/cookbook/cookbook.htm Priya Sharma Trivedi JUnit: Testing code as you write it Thomas Zimmermann Unit Testing with JUnit

Thank you