Coverage

completely\/exhaustively testing a system is nearly impossible within practical time boundary

includes all of underlying code (statements, loops, branches)

• heuristics applied instead to fold input space into different functional categories THEN create tests that sample behavior\/output for each functional partition

example:

Multi Level Testing

Functional - Black Box does system behave as predicted by specification?

Structural - White Box are all code aspects are covered? statements, branches, loops?

Black Box

Focus: I\/O behavior (given input, we can produce predicted output)

Almost always impossible to generate all possible inputs (test cases) SO Reduce number of test cases by equivalence partitioning:

1. Divide input conditions into equivalence classes

2. Choose test cases for each equivalence class.

   1. i.e. if an object is supposed to accept a negative number, testing one negative number is enough

Categories: Fucntionality:

• user input verification (based on specification)

• output results

• state transitions

  • are there clear states in the system in which the system is supposed to behave differently based on the state?
  • boundary cases

Grey Box

Use knowledge of system’s architecture to create a more complete set of black box tests

• Verifying auditing and logging information for each Fn (is the system really updating all internal state correctly?)

• Data destined for other systems

• System-added information (timestamps, checksums, etc.)

• “Looking for Scraps”

  • Is the system correctly cleaning up after itself?

    • temporary files, memory leaks, data duplication\/deletion

White Box

Explicit knowledge of the internal workings of the system being tested are used to select the test data

Control Flow Graph (CFG)

Program Graph: program written in an imperative programming language. A directed labeled graph in which:

• Nodes individual statements OR groups of entire statements

• Edges flow of control

There is an edge from node i, to node j _in the program graph IF AND ONLY IF , the statement corresponding to node j, _can be executed immediately _after the last statement that correspond to node _i

Code Coverage

Writing test cases with complete knowledge of code. Format = same: input, expected output, actual output. Includes following criteria for completeness:

• Statement Testing: write tests until all statements have been executed

• Edge(branch) Testing: write tests until each edge in a program’s control flow graph has been executed at least once (covers true\/false conditions)

• Condition Testing: like edge coverage but with additional attention paid to the conditionals - if compound conditional, ensure that all combinations have been covered

• Path testing: path includes all nodes from the initial to the final node - all paths in a program’s control flow graph have been executed multiple times as dictated by heuristics, e.g.,

  • for each loop-based path,write a test case that executes the loop

    • 0 times (skips the loop)

    • 1 one time

    • >1 (exact number depends on context)

Code Coverage Tools: tools that can track code coverage metrics

• typically just statement and branch coverage

• generate reports on % of metric achieved

• typically provide a view of the source code annoted to show which statements and conditions were "hit" by the test suite

Testing Automation: makes testing easy (run tests with single command)

• often part of the buils management process