Boost Test Library Design

Definitions
Requirements
Experience
Layered Design
Factored Design
User-supplied Components
Usage Patterns
Paths Not Taken

Definitions

Execution Tools - Reports general errors from lower-level components, and unifies various general error notifications (return codes, exceptions, etc.) into the form appropriate for the host operating environment. Intended for use with both production and test programs.

Test Tools - Execution Tools features, plus reports test errors from lower-level components, and unifies various test error notifications (test library specific return codes, exceptions, etc.) into the form appropriate for the caller.  Functions or macros easing test case construction.  Intended for use primarily in test programs.

Unit Test Tools - Provides a framework which facilitates the testing of C++ classes, and the organization of these tests into test-suites. Intended for use almost exclusively in test programs.

Requirements

Requirement Execution
Tools
Test
Tools
Unit Test
Tools
Report errors from lower-level components. required required required
Unify error notifications (return codes, exceptions, etc.) into a form appropriate for the host operating environment. required required required
Virtually no changes required in existing user programs.  Renaming main() allowed, but that's about it.  (Note that changing existing programs may be desirable to take advantage of certain features. The point of this requirement is that you don't have to if you don't want to.) required required not required; most programs using unit test tools will be written from scratch to do so.
Virtually no perceived coupling or dependency costs. required required some cost acceptable
Maximum time required for an intermediate programmer to learn enough to successfully use the tool. 10 seconds plus however long it takes to convince themselves renaming main() is really all they have to do. 5-10 minutes 30-60 minutes 
Functions or macros available to ease test case construction. not required or even desirable required required
Classes, functions or macros to ease test suite construction. not required or even desirable not required or even desirable required

Experience

Because the requirements for the other tools are almost a subset of the Unit Test Tools requirements, it is tempting to supply only the Unit Test Tools, perhaps with some convenience functions to ease use for simple test or execution error-detection.  Likewise, since Test Tools supplies the functionality of Execution Tools, the Test Tools could be used Execution Toos.

Both unified approaches have been rejected, however, because of past negative experience:

Layered Design

The Main concept is equivalent to the specifications for a C++ main() function as defined by the ISO C++ Standard.

The Execution Tools provides a non-recursive program startup function which calls a user-supplied cpp_main() function which meets the Main concept requirements. The supplied default implementation for standard C++ environments uses main() for startup.  A GUI implementation might supply a different start-up function, such as WinMain(), but all implementations require the user supply a cpp_main() function.  Note that the cpp_main() function may be supplied by the Test Tools or by a user written program.

The startup function (that is, the replacement main()) shall meet its notification and unification requirements above, including handling of all standard library exceptions, char * and string exceptions, and if possible, signals from the operating system or the hardware.

The Test Tools provide a cpp_main() function which calls a user-supplied test_main() function, both of which meet the Main concept requirements. Note that the test_main() function may be supplied by the Unit Test Tools or by a user written program.

The Test Tools also provide a (independent, no other header dependencies) header containing convenience macros and function declarations.

The cpp_main() function for Test Tools shall meet its notification and unification requirements above.  The Test Tools implementation supplies the definitions for the functions supplied by the test_tools header.

Unit Test Tools shall provide a test_main() function which meets the Main concept requirements.

A layered design is chosen as meeting all the requirements, being very flexible and easy to implement, eliminating coupling and dependencies between the libraries, allowing swapping of implementations, and allowing unit test libraries to be developed at a later date.

The  Execution Tools and Test Tools both offer an unusual choice of allowing the user to choose either a compile-time implementation by treating the implementation files as headers, or a link-time implementation by treating the same implementation files as separately compiled library source files.  (This portion of the design was contributed by Ed Brey.)

Factored Design

The code which catches exceptions has been factored out (see boost/detail/catch_exceptions.hpp).  This is done to facilitate reuse - this capability may be useful in other test libraries or frameworks.  

User-supplied Components

In addition to being concrete implementations, the libraries act as a set of specifications for their components.  User-supplied components which meet those specifications may replace the library supplied components.

Usage Patterns

The expected usage patterns include:

Paths Not Taken

The Main concept is the one used by the C++ Standard, as inherited from the C language.  More modern approaches were considered, including a generic programming design parameterized on a function object type, and an object-oriented design based on a common execution object.  Both were rejected as maximal solutions to a minimal need, and as not meeting the "virtually no change in user programs" requirement.

In the case of needing to make additional GUI arguments to startup functions available, the requirement that startup function be non-recursive was added.  This makes it simple and safe for the startup function source file to store such arguments in its unnamed namespace, and supply an operating system dependent function in the same source file to access those arguments.


© Beman Dawes 2001

Revised: 09 March, 2001