Test design

In software engineering, test design is the activity of deriving and specifying test cases from test conditions to test software.

Definition

A test condition is a statement about the test object. Test conditions can be stated for any part of a component or system that could be verified: functions, transactions, features, quality attributes or structural elements.

The fundamental challenge of test design is that there are infinitely many different tests that you could run, but there is not enough time to run them all. A subset of tests must be selected; small enough to run, but well-chosen enough that the tests find bug and expose other quality-related information.

Test design is one of the most important prerequisites of software quality. Good test design supports:

1. defining and improving quality related processes and procedures (quality assurance);
2. evaluating the quality of the product with regards to customer expectations and needs (quality control);
3. finding defects in the product (software testing).

The essential prerequisites of test design are:

1. Appropriate specification (test bases).
2. Risk and complexity analysis.
3. Historical data of your previous developments (if exists).

The test bases, such as requirements or user stories, determine what should be tested (test objects and test conditions). The test bases involves some test design techniques to be used or not to be used.

Risk analysis is inevitable to decide the thoroughness of testing. The more risk the usage of the function/object has, the more thorough the testing that is needed. The same can be said for complexity. Risk and complexity analysis determines the test design techniques to be applied for a given specification.

Historical data of your previous developments help setting the best set of test design techniques to reach a cost optimum and high quality together. In lack of historical data some assumptions can be made, which should be refined for subsequent projects.

Based on these prerequisites an optimal test design strategy can be implemented.

The result of the test design is a set of test cases based on the specification. These test cases can be designed prior to the implementation starts, and should be implementation-independent. Test first way of test design is very important as efficiently supports defect prevention. Based on the application and the present test coverage further test cases can be created (but it is not test design).

In practice, more test design techniques should be applied together for complex specifications.

Altogether, test design does not depend on the extraordinary (near magical) skill of the person creating the test but is based on well understood principles.

ISO/IEC/IEEE 29119-4:2015, Part 4 details the standard definitions of test design techniques. The site of test designers offers the LEA (Learn-Exercise-Apply) methodology to support effective learning, exercising and applying the techniques.

Automatic test design

Entire test suites or test cases exposing real bugs can be automatically generated by software using model checking or symbolic execution. Model checking can ensure all the paths of a simple program are exercised, while symbolic execution can detect bugs and generate a test case that will expose the bug when the software is run using this test case.

However, as good as automatic test design can be, it is not appropriate for all circumstances. If the complexity becomes too high, then human test design must come into play as it is far more flexible and it can concentrate on generating higher level test suites.