UTF"> ]>

Testing tools

The &utf;'s supplies a toolbox of testing tools to ease creation and maintenance of test programs and provide a uniform error reporting mechanism. The toolbox supplied in most part in a form of macro and function declarations. While the functions can be called directly, the usual way to use testing tools is via convenience macros. All macros arguments are calculated once, so it's safe to pass complex expressions in their place. All tools automatically supply an error location: a file name and a line number. The testing tools are intended for unit test code rather than library or production code, where throwing exceptions, using assert(), boost::concept_check or BOOST_STATIC_ASSERT() may be more suitable ways to detect and report errors. For list of all supplied testing tools and usage examples see the reference.

All the tools are supplied in three flavors(levels): WARN, CHECK and REQUIRE. For example: BOOST_WARN_EQUAL, BOOST_CHECK_EQUAL, BOOST_REQUIRE_EQUAL. If an assertion designated by the tool passes, confirmation message can be printed in log outputto manage what messages appear in the test log stream set the proper log level. If an assertion designated by the tool failed, depending on the level following will happenedin some cases log message can be slightly different to reflect failed tool specifics: Level Test log content Errors counter Test execution WARN warning in <test case name>: condition <assertion description> is not satisfied not affected continues CHECK error in <test case name>: test <assertion description> failed increased continues REQUIRE fatal error in <test case name>: critical test <assertion description> failed increased aborts Regularly you should use CHECK level tools to implement your assertions. You can use WARN level tools to validate aspects less important then correctness: performance, portability, usability etc. You should use REQUIRE level tools only if continuation of the test case doesn't make sense if this assertions fails.

How do you perform correctness test for operator<<( std::ostream&, ... ) operations? You can print into the standard output stream and manually check that it is matching your expectations. Unfortunately, this is not really acceptable for the regression testing and doesn't serve a long term purpose of a unit test. You can use std::stringstream and compare resulting output buffer with the expected pattern string, but you are required to perform several additional operations with every check you do. So it becomes tedious very fast. The class output_test_stream is designed to automate these tasks for you. This is a simple, but powerful tool for testing standard std::ostream based output operation. The class output_test_stream complies to std::ostream interface so it can be used in place of any std::ostream parameter. It provides several test methods to validate output content, including test for match to expected output content or test for expected output length. Flushing, synchronizing, string comparison and error message generation is automated by the tool implementation. All output_test_stream validation methods by default flush the stream once check is performed. If you want to perform several checks with the same output, specify parameter flush_stream with value false. This parameter is supported on all comparison methods. In some cases manual generation of expected output is either too time consuming or is impossible at all bacause of sheer volume. What we need in cases like this is to be able to check once manually that the output is as expected and to be able in a future check that it stays the same. To help manage this logic the class output_test_stream allows matching output content versus specified pattern file and produce pattern file based on successful test run. Detailed specification of class output_test_stream is covered in reference section.

There are two ways to employ the class output_test_stream: explicit output checks and pattern file matching.

Use the instance of class output_test_stream as an output stream and check output content using tool's methods. Note use of false to prevent output flushing in first two invocation of check functions. Unless you want to perform several different checks for the same output you wouldn't need to use it though. Your test will look like a serious of output operators followed by one check. And so on again. Try to perform checks as frequently as possible. It not only simplifies patterns you compare with, but also allows you to more closely identify possible source of failure. Even simpler: no need to generate expected patterns. Though you need to keep the pattern file all the time somewhere around. Your testing will look like a serious of output operators followed by match pattern checks repeated several times. Try to perform checks as frequently as possible, because it allows you to more closely identify possible source of failure. Content of the pattern file is: i=2 File: test.cpp Line: 14

Even though supplied testing tools cover wide range of possible checks and provide detailed report on cause of error in some cases you may want to implement and use custom predicate that perform complex check and produce intelligent report on failure. To satisfy this need testing tools implement custom predicate support. There two layers of custom predicate support implemented by testing tools toolbox: with and without custom error message generation. The first layer is supported by BOOST_CHECK_PREDICATE family of testing tools. You can use it to check any custom predicate that reports the result as boolean value. The values of the predicate arguments are reported by the tool automatically in case of failure. To use second layer your predicate have to return boost::test_tools::predicate_result. This class encapsulates boolean result value along with any error or information message you opt to report. Usually you construct the instance of class boost::test_tools::predicate_result inside your predicate function and return it by value. The constructor expects one argument - the boolean result value. The constructor is implicit, so you can simply return boolean value from your predicate and boost::test_tools::predicate_result is constructed automatically to hold your value and empty message. You can also assign boolean value to the constructed instance. You can check the current predicate value by using operator!() or directly accessing public read-only property p_predicate_value. The error message is stored in public read-write property p_message.

In most cases it is unreasonable to use an operator==(...) for a floating-point values equality check. The simple, absolute value comparison based, solution for a floating-point values u, v and a tolerance &egr;: |u − v| ≤ &egr; does not produce expected results in many circumstances - specifically for very small or very big values (See for examples). The &utf; implements floating-point comparison algorithm that is based on the more confident solution first presented in : |u − v| ≤ &egr; × |u| ∧ |u − v| ≤ &egr; × |v| defines a very close with tolerance &egr; relationship between u and v |u − v| ≤ &egr; × |u| ∨ |u − v| ≤ &egr; × |v| defines a close enough with tolerance &egr; relationship between u and v Both relationships are commutative but are not transitive. The relationship defined by inequations (2) is stronger that the relationship defined by inequations (3) (i.e. (2) ⇒ (3)). Because of the multiplication in the right side of inequations, that can cause an unwanted underflow condition, the implementation is using modified version of the inequations (2) and (3) where all underflow, overflow conditions can be guarded safely: |u − v| / |u| ≤ &egr; ∧ |u − v| / |v| ≤ &egr; |u − v| / |u| ≤ &egr; ∨ |u − v| / |v| ≤ &egr; Checks based on equations (4) and (5) are implemented by two predicates with alternative interfaces: binary predicate close_at_tolerancecheck type and tolerance value are fixed at predicate construction time and predicate with four arguments check_is_closecheck type and tolerance value are the arguments of the predicate. While equations (4) and (5) in general are preferred for the general floating point comparison check over equation (1), they are unusable for the test on closeness to zero. The later check is still might be useful in some cases and the &utf; implements an algorithm based on equation (1) in binary predicate check_is_smallv is zero. On top of the generic, flexible predicates the &utf; implements macro based family of tools BOOST_CHECK_CLOSE and BOOST_CHECK_SMALL. These tools limit the check flexibility to strong-only checks, but automate failed check arguments reporting.

In case of absence of domain specific requirements the value of tolerance can be chosen as a sum of the predicted upper limits for "relative rounding errors" of compared values. The "rounding" is the operation by which a real value 'x' is represented in a floating-point format with 'p' binary digits (bits) as the floating-point value 'X'. The "relative rounding error" is the difference between the real and the floating point values in relation to real value: |x-X|/|x|. The discrepancy between real and floating point value may be caused by several reasons: Type promotion Arithmetic operations Conversion from a decimal presentation to a binary presentation Non-arithmetic operation The first two operations proved to have a relative rounding error that does not exceed ½ × "machine epsilon value" for the appropriate floating point type (represented by std::numeric_limits<FPT>::epsilon()). Conversion to binary presentation, sadly, does not have such requirement. So we can't assume that float 1.1 is close to real 1.1 with tolerance ½ × "machine epsilon value" for float (though for 11./10 we can). Non arithmetic operations either do not have a predicted upper limit relative rounding errors. Note that both arithmetic and non-arithmetic operations might also produce others "non-rounding" errors, such as underflow/overflow, division-by-zero or 'operation errors'. All theorems about the upper limit of a rounding error, including that of ½ × epsilon, refer only to the 'rounding' operation, nothing more. This means that the 'operation error', that is, the error incurred by the operation itself, besides rounding, isn't considered. In order for numerical software to be able to actually predict error bounds, the IEEE754 standard requires arithmetic operations to be 'correctly or exactly rounded'. That is, it is required that the internal computation of a given operation be such that the floating point result is the exact result rounded to the number of working bits. In other words, it is required that the computation used by the operation itself doesn't introduce any additional errors. The IEEE754 standard does not require same behavior from most non-arithmetic operation. The underflow/overflow and division-by-zero errors may cause rounding errors with unpredictable upper limits. At last be aware that ½ × epsilon rules are not transitive. In other words combination of two arithmetic operations may produce rounding error that significantly exceeds 2 × ½ × epsilon. All in all there are no generic rules on how to select the tolerance and users need to apply common sense and domain/ problem specific knowledge to decide on tolerance value. To simplify things in most usage cases latest version of algorithm below opted to use percentage values for tolerance specification (instead of fractions of related values). In other words now you use it to check that difference between two values does not exceed x percent. For more reading about floating-point comparison see references below.

KnuthII Donald. E.Knuth 1998Addison-Wesley Longman, Inc. 0-201-89684-2 Addison-Wesley Professional; 3 edition Kulisch Ulrich WKulisch 2002Springer, Inc. 0-201-89684-2 Springer; 1 edition Squassabia AlbertoSquassabia March 2000. Becker PeteBecker December 2000. Goldberg DavidGoldberg 1991Association for Computing Machinery, Inc. 150-230 March. Langlois PhilippeLanglois 2000 0249-6399 Kahan WilliamKahan

Reference These tools are used to validate the predicate value. The only parameter for these tools is a boolean predicate value that gets validated. It could be any expression that could be evaluated and converted to boolean value. The expression gets evaluated only once, so it's safe to pass complex expression for validation. BOOST_<level>_MESSAGE These tools are used to perform bitwise comparison of two values. The check shows all positions where left and right value's bits mismatch. The first parameter is the left compared value. The second parameter is the right compared value. Parameters are not required to be of the same type, but warning is issued if their type's size does not coincide. BOOST_<level>_EQUAL These tools are used to check on closeness using strong relationship defined by the predicate check_is_close( left, right, tolerance ). To check for the weak relationship use BOOST_<level>_PREDICATE family of tools with explicit check_is_close invocation. The first parameter is the left compared value. The second parameter is the right compared value. Last third parameter defines the tolerance for the comparison in percentage units. It is required for left and right parameters to be of the same floating point type. You will need to explicitly resolve any type mismatch to select which type to use for comparison. Note that to use these tools you need to include additional header floating_point_comparison.hpp. BOOST_<level>_CLOSE_FRACTION, BOOST_<level>_SMALL, BOOST_<level>_EQUAL, Floating point comparison algorithms These tools are used to check on closeness using strong relationship defined by the predicate check_is_close( left, right, tolerance ). To check for the weak relationship use BOOST_<level>_PREDICATE family of tools with explicit check_is_close invocation. The first parameter is the left compared value. The second parameter is the right compared value. Last third parameter defines the tolerance for the comparison as fraction of absolute values being compared. It is required for left and right parameters to be of the same floating point type. You will need to explicitly resolve any type mismatch to select which type to use for comparison. Note that to use these tools you need to include additional header floating_point_comparison.hpp. BOOST_<level>_CLOSE, BOOST_<level>_SMALL, BOOST_<level>_EQUAL, Floating point comparison algorithms Check performed by these tools is the same as the one performed by BOOST_<level>( left == right ). The difference is that the mismatched values are reported as well. It is bad idea to use these tools to compare floating point values. Use BOOST_<level>_CLOSE or BOOST_<level>_CLOSE_FRACTION tools instead. BOOST_<level>, BOOST_<level>_CLOSE, BOOST_<level>_NE These tools are used to perform an element by element comparison of two collections. They print all mismatched positions, collection elements at these positions and check that the collections have the same size. The first two parameters designate begin and end of the first collection. The two parameters designate begin and end of the second collection. BOOST_<level>_EQUAL These tools are used to perform an exception detection and validation check. Tools execute the supplied expression and validate that it throws an exception of supplied class (or the one derived from it) that complies with the supplied predicate. If the expression throws any other unrelated exception, doesn't throw at all or predicate evaluates to false, check fails. In comparison with BOOST_<level>_THROW tools these allow performing more fine-grained checks. For example: make sure that an expected exception has specific error message. BOOST_<level>_THROW Check performed by these tools is the same as the one performed by BOOST_<level>( left >= right ). The difference is that the argument values are reported as well. BOOST_<level>_LE, BOOST_<level>_LT, BOOST_<level>_GT Check performed by these tools is the same as the one performed by BOOST_<level>( left > right ). The difference is that the argument values are reported as well. BOOST_<level>_LE, BOOST_<level>_LT, BOOST_<level>_GE Check performed by these tools is the same as the one performed by BOOST_<level>( left <= right ). The difference is that the argument values are reported as well. BOOST_<level>_LT, BOOST_<level>_GE, BOOST_<level>_GT Check performed by these tools is the same as the one performed by BOOST_<level>( left < right ). The difference is that the argument values are reported as well. BOOST_<level>_LE, BOOST_<level>_GE, BOOST_<level>_GT These tools perform exactly the same check as BOOST_<level> tools. The only difference is that instead of generating an error/confirm message these use the supplied one. The first parameter is the boolean expression. The second parameter is the message reported in case of check failure. The message argument can be constructed of components of any type supporting the std::ostream& operator<<(std::ostream&). BOOST_<level> Check performed by these tools is the same as the one performed by BOOST_<level>( left != right ). The difference is that the matched values are reported as well. BOOST_<level>_EQUAL These tools are used to perform a "no throw" check. Tools execute the supplied expression and validate that it does not throw any exceptions. Error would be reported by the framework even if the statement appear directly in test case body and throw any exception. But these tools allow proceeding further with test case in case of failure. If check is successful, tools may produce a confirmation message, in other case they produce an error message in a form "error in <test case name>exception was thrown by <expression>. The only parameter is an expression to execute. You can use do-while(0) block if you want to execute more than one statement. BOOST_<level>_THROW These are generic tools used to validate an arbitrary supplied predicate functor (there is a compile time limit on predicate arity defined by the configurable macro BOOST_TEST_MAX_PREDICATE_ARITY). To validate zero arity predicate use BOOST_<level> tools. In other cases prefer theses tools. The advantage of these tools is that they show arguments values in case of predicate failure. The first parameter is the predicate itself. The second parameter is the list of predicate arguments each wrapped in round brackets (BOOST_PP sequence format). Note difference in error log from BOOST_<level> These tools are used to check that supplied value is small enough. The "smallness" is defined by absolute value of the tolerance supplied as a second argument. Use these tools with caution. To compare to values on closeness it's preferable to use BOOST_<level>_CLOSE tools instead. The first parameter is the value to check. The second parameter is the tolerance. Note that to use these tools you need to include additional header floating_point_comparison.hpp. BOOST_<level>_CLOSE, BOOST_<level>_CLOSE_FRACTION, Floating point comparison algorithms These tools are used to perform an exception detection check. Tools execute the supplied expression and validate that it throws an exception of supplied class (or the one derived from it) or it's child. If the statement throws any other unrelated exception or doesn't throw at all, check fails. If check is successful, the tool produces a confirmation message, in other case it produces an error message in a form "error in test case name: exception exception expected. The first parameter is the expression to execute. Use do-while(0) block if you want to execute more than one statement. The second parameter is an expected exception. BOOST_<level>NO_THROW BOOST_ERROR tool behave the same way as BOOST_CHECK_MESSAGE( false, message ). This tool is used for an unconditional error counter increasing and message logging. The only tool's parameter is an error message to log. BOOST_<level> BOOST_FAIL behave the same way as BOOST_REQUIRE_MESSAGE( false, message ). This tool is used for an unconditional error counter increasing, message logging and the current test case aborting. The only tool's parameter is an error message to log. BOOST_<level> Unlike the rest of the tools in the toolbox this tool does not perform the logging itself. Its only purpose is to check at runtime whether or not the supplied preprocessor symbol is defined. Use it in combination with BOOST_<level> to perform and log validation. Macros of any arity could be checked. To check the macro definition with non-zero arity specify dummy arguments for it. See below for example. The only tool's parameter is a preprocessor symbol that gets validated. BOOST_<level>