mirror of
https://github.com/boostorg/test.git
synced 2026-02-18 14:32:11 +00:00
858 lines
37 KiB
XML
858 lines
37 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
|
<!DOCTYPE section PUBLIC "-//Boost//DTD BoostBook XML V1.0//EN" "../../../../tools/boostbook/dtd/boostbook.dtd" [
|
|
<!ENTITY utf "<acronym>UTF</acronym>">
|
|
]>
|
|
<section id="utf.user-guide.test-organization">
|
|
<title>Test organization … or the house that Jack built</title>
|
|
<titleabbrev>Test organization</titleabbrev>
|
|
|
|
<para role="first-line-indented">
|
|
If you look at many legacy test modules, big chance is that it's implemented as one big test function that
|
|
consists of a mixture of check and output statements. Is there anything wrong with it? Yes. There are various
|
|
disadvantages in single test function approach:
|
|
</para>
|
|
|
|
<itemizedlist mark="square">
|
|
<listitem>
|
|
<simpara>
|
|
One big function tends to become really difficult to manage if the number of checks exceeds a reasonable limit
|
|
(true for any large function). What is tested and where - who knows?
|
|
</simpara>
|
|
</listitem>
|
|
<listitem>
|
|
<simpara>
|
|
Many checks require similar preparations. This results in code repetitions within the test function.
|
|
</simpara>
|
|
</listitem>
|
|
<listitem>
|
|
<simpara>
|
|
If a fatal error or an exception is caused by any checks within the test function the rest of tests are
|
|
skipped and there is no way to prevent this.
|
|
</simpara>
|
|
</listitem>
|
|
<listitem>
|
|
<simpara>
|
|
No way to perform only checks for a particular subsystem of the tested unit.
|
|
</simpara>
|
|
</listitem>
|
|
<listitem>
|
|
<simpara>
|
|
No summary of how different subsystems of the tested unit performed under in the test.
|
|
</simpara>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para role="first-line-indented">
|
|
The above points should make it clear that it's preferable to split <link linkend="test-module.def">test module
|
|
</link> into smaller units. These units are test cases. A test case has to be constructed based on some kind of
|
|
function and registered in a test tree, so that the test runner knows how to invoke it. There are different
|
|
possible designs for the test case construction problem: inheritance from the predefined base class, specifically
|
|
named member function and so on. The &utf; opted to avoid classed altogether and to use the least intrusive "
|
|
generic callback" approach. The facility, the &utf; provides, requires specific function signature and allows
|
|
adopting any function or function object that matches the signature as a test case. The signatures the &utf;
|
|
supports are:
|
|
</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<simpara>
|
|
<link linkend="utf.user-guide.test-organization.nullary-test-case">Nullary function</link>
|
|
</simpara>
|
|
</listitem>
|
|
<listitem>
|
|
<simpara>
|
|
<link linkend="utf.user-guide.test-organization.unary-test-case">Unary function</link>
|
|
</simpara>
|
|
</listitem>
|
|
<listitem>
|
|
<simpara>
|
|
<link linkend="utf.user-guide.test-organization.test-case-template">Nullary function template</link>
|
|
</simpara>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para role="first-line-indented">
|
|
To solve test tree creation problem the &utf; provides facilities for
|
|
<link linkend="utf.user-guide.test-organization.test-suite">test suite creation</link>.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
Generic test case construction design used by the &utf; causes the test case implementation (test function body)
|
|
and test case creation/registration points to be remote. As a result you may forget to register the test case
|
|
and it's never going to be executed, even though it's present in test file. To alleviate this issue
|
|
the &utf; presents facilities for automated (in place) test case creation and registration in the test tree. These
|
|
facilities sacrifice some generality and work for selected test function signatures only. But the result is that
|
|
library users are relieved from the necessity to manually register test cases. These facilities are the most
|
|
user-friendly and are recommended to be used whenever possible. In addition they support automated registration
|
|
of test suites, thus allowing whole test tree to be created without any use of manual registration.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
The single test module may mix both automated and manual test case
|
|
registration. In other words, within the same test module you can have both test cases implemented remotely and
|
|
registered manually in the test module initialization function and test cases that are registered automatically at
|
|
implementation point.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
In some cases it's desirable to allow some "expected" failures in test case without failing a
|
|
test module. To support this request The &utf; allows specifying the number of
|
|
<link linkend="utf.user-guide.test-organization.expected-failures">expected failures</link> in a test case.
|
|
</para>
|
|
|
|
<section id="utf.user-guide.test-organization.nullary-test-case">
|
|
<title>Nullary function based test case</title>
|
|
|
|
<para role="first-line-indented">
|
|
The most widely used are test cases based on a nullary function. These include nullary free functions, nullary
|
|
function objects created with <functionname>boost::bind</functionname> and nullary <classname>boost::function</classname>
|
|
instances. The simplest is a free function and the &utf; provides facilities to create a free function based test
|
|
case that is automatically registered. Here are the two construction interfaces:
|
|
</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<simpara>
|
|
<link linkend="utf.user-guide.test-organization.auto-nullary-test-case">Test case with automated registration</link>
|
|
</simpara>
|
|
</listitem>
|
|
<listitem>
|
|
<simpara>
|
|
<link linkend="utf.user-guide.test-organization.manual-nullary-test-case">Manually registered test case</link>
|
|
</simpara>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<section id="utf.user-guide.test-organization.auto-nullary-test-case">
|
|
<title>Nullary function based test case with automated registration</title>
|
|
<titleabbrev>Automated registration</titleabbrev>
|
|
|
|
<para role="first-line-indented">
|
|
To create a nullary free function cased test case, which is registered in place of implementation, employ the
|
|
macro BOOST_AUTO_TEST_CASE.
|
|
</para>
|
|
|
|
<inline-synopsis>
|
|
<macro name="BOOST_AUTO_TEST_CASE" kind="functionlike">
|
|
<macro-parameter name="test_case_name"/>
|
|
</macro>
|
|
</inline-synopsis>
|
|
|
|
<para role="first-line-indented">
|
|
The macro is designed to closely mimic nullary free function syntax. Changes that are required to make an
|
|
existing test case, implemented as a free function, registered in place are illustrated in the following
|
|
example (compare with <xref linkend="utf.user-guide.test-organization.manual-nullary-test-case.example01"/>):
|
|
</para>
|
|
|
|
<btl-example name="example06">
|
|
<title>Nullary function based test case with automated registration</title>
|
|
</btl-example>
|
|
|
|
<para role="first-line-indented">
|
|
With this macro you don't need to implement the initialization function at all. The macro creates and
|
|
registers the test case with the name free_test_function automatically.
|
|
</para>
|
|
</section>
|
|
|
|
<section id="utf.user-guide.test-organization.manual-nullary-test-case">
|
|
<title>Manually registered nullary function based test case</title>
|
|
<titleabbrev>Manual registration</titleabbrev>
|
|
|
|
<para role="first-line-indented">
|
|
To create a test case manually, employ the macro BOOST_TEST_CASE:
|
|
</para>
|
|
|
|
<inline-synopsis>
|
|
<macro name="BOOST_TEST_CASE" kind="functionlike">
|
|
<macro-parameter name="test_function"/>
|
|
</macro>
|
|
</inline-synopsis>
|
|
|
|
<para role="first-line-indented">
|
|
BOOST_TEST_CASE creates an instance of the class boost::unit_test::test_case and returns a pointer to the
|
|
constructed instance. The test case name is deduced from the macro argument test_function. If you prefer to
|
|
assign a different test case name, you have to use the underlying make_test_case interface instead. To
|
|
register a new test case, employ the method test_suite::add. Both test case creation and registration are
|
|
performed in the test module initialization function.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
Here is the simplest possible manually registered test case. This example employs the original test module
|
|
initialization function specification. A single test case is created and registered in the master test suite.
|
|
Note that the free function name is passed by address to the macro BOOST_TEST_CASE.
|
|
</para>
|
|
|
|
<btl-example name="example01">
|
|
<title>Nullary free function manually registered</title>
|
|
</btl-example>
|
|
|
|
<para role="first-line-indented">
|
|
A test case can be implemented as a method of a class. In this case a pointer to the class instance has to be
|
|
bound to the test method to create a test case. You can use the same instance of the class for multiple test
|
|
cases. The &utf; doesn't take an ownership of the class instance and you are required to manage the class
|
|
instance lifetime yourself.
|
|
</para>
|
|
|
|
<warning>
|
|
<simpara>
|
|
The class instance can't be defined in the initialization function scope, since it becomes invalid as
|
|
soon as the test execution exits it. It needs to be either defined statically/globally or managed using a
|
|
shared pointer.
|
|
</simpara>
|
|
</warning>
|
|
|
|
<btl-example name="example02">
|
|
<title>Nullary method of a class bound to global class instance and manually registered</title>
|
|
</btl-example>
|
|
|
|
<btl-example name="example03">
|
|
<title>Nullary method of a class bound to shared class instance and manually registered</title>
|
|
</btl-example>
|
|
|
|
<para role="first-line-indented">
|
|
If you do not need to reuse the test class instance and can't or do not wish to create test class
|
|
instance globally it may be easier and safer to create an instance on the stack of free function:
|
|
</para>
|
|
|
|
<btl-example name="example04">
|
|
<title>Nullary method of a class bound to local class instance inside free function</title>
|
|
</btl-example>
|
|
|
|
<para role="first-line-indented">
|
|
If you have to perform the same set of tests with different sets of parameters you may want to base your test
|
|
case on a function with arguments and bind particular parameters during test case creation.
|
|
</para>
|
|
|
|
<warning>
|
|
<simpara>
|
|
If you bind parameters by reference or pointer, the referenced value can't have local storage in the
|
|
test module initialization function.
|
|
</simpara>
|
|
</warning>
|
|
|
|
<btl-example name="example05">
|
|
<title>Binary free function bound to set of different parameter pairs</title>
|
|
|
|
<para role="first-line-indented">
|
|
This example employs the alternative test module initialization function specification.
|
|
</para>
|
|
</btl-example>
|
|
|
|
<para role="first-line-indented">
|
|
The &utf; also presents an alternative method for parameterized test case creation, which is covered in
|
|
<xref linkend="utf.user-guide.test-organization.unary-test-case"/>.
|
|
</para>
|
|
</section>
|
|
</section>
|
|
<section id="utf.user-guide.test-organization.unary-test-case">
|
|
<title>Unary function based test case</title>
|
|
|
|
<para role="first-line-indented">
|
|
Some tests are required to be repeated for a series of different input parameters. One way to achieve this is
|
|
manually register a test case for each parameter as in example above. You can also invoke a test function with
|
|
all parameters manually from within your test case, like this:
|
|
</para>
|
|
|
|
<btl-snippet name="snippet1"/>
|
|
|
|
<para role="first-line-indented">
|
|
The &utf; presents a better solution for this problem: the unary function based test case, also referred as
|
|
parameterized test case. The unary test function can be a free function, unary functor (for example created
|
|
with boost::bind) or unary method of a class with bound test class instance). The test function is converted
|
|
into test case using the macro BOOST_PARAM_TEST_CASE. The macro expects a collection of parameters (passed as
|
|
two input iterators) and an unary test function:
|
|
</para>
|
|
|
|
<inline-synopsis>
|
|
<macro name="BOOST_PARAM_TEST_CASE" kind="functionlike">
|
|
<macro-parameter name="test_function"/>
|
|
<macro-parameter name="params_begin"/>
|
|
<macro-parameter name="params_end"/>
|
|
</macro>
|
|
</inline-synopsis>
|
|
|
|
<para role="first-line-indented">
|
|
BOOST_PARAM_TEST_CASE creates an instance of the test case generator. When passed to the method test_suite::add,
|
|
the generator produces a separate sub test case for each parameter in the parameters collection and registers
|
|
it immediately in a test suite. Each test case is based on a test function with the parameter bound by value,
|
|
even if the test function expects a parameter by reference. The fact that parameter value is stored along with
|
|
bound test function releases you from necessity to manage parameters lifetime. For example, they can be defined
|
|
in the test module initialization function scope.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
All sub test case names are deduced from the macro argument test_function. If you prefer to assign different
|
|
names, you have to use the underlying make_test_case interface instead. Both test cases creation and
|
|
registration are performed in the test module initialization function.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
The parameterized test case facility is preferable to the approach in the example above, since execution of
|
|
each sub test case is guarded and counted independently. It produces a better test log/results report (in
|
|
example above in case of failure you can't say which parameter is at fault) and allows you to test against
|
|
all parameters even if one of them causes termination a particular sub test case.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
In comparison with a manual test case registration for each parameter approach the parameterized test case
|
|
facility is more concise and easily extendible.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
In following simple example the same test, implemented in <code>free_test_function</code>, is
|
|
performed for 5 different parameters. The parameters are defined in the test module initialization function
|
|
scope. The master test suite contains 5 independent test cases.
|
|
</para>
|
|
|
|
<btl-example name="example07">
|
|
<title>Unary free function based test case</title>
|
|
</btl-example>
|
|
|
|
<para role="first-line-indented">
|
|
Next example is similar, but instead of a free function it uses a method of a class. Even though parameters are
|
|
passed into test method by reference you can still define them in the test module initialization function scope.
|
|
This example employs the alternative test module initialization function specification.
|
|
</para>
|
|
|
|
<btl-example name="example08">
|
|
<title>Unary class method based test case</title>
|
|
</btl-example>
|
|
</section>
|
|
|
|
<section id="utf.user-guide.test-organization.test-case-template">
|
|
<title>Test case template</title>
|
|
<para role="first-line-indented">
|
|
To test a template based component it's frequently necessary to perform the same set of checks for a
|
|
component instantiated with different template parameters. The &utf; provides the ability to create a series of
|
|
test cases based on a list of desired types and function similar to nullary function template. This facility is
|
|
called test case template. Here are the two construction interfaces:
|
|
</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<simpara>
|
|
<link linkend="utf.user-guide.test-organization.auto-test-case-template">Test case template with automated
|
|
registration</link>
|
|
</simpara>
|
|
</listitem>
|
|
<listitem>
|
|
<simpara>
|
|
<link linkend="utf.user-guide.test-organization.manual-test-case-template">Manually registered test case
|
|
template</link>
|
|
</simpara>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<section id="utf.user-guide.test-organization.auto-test-case-template">
|
|
<title>Test case template with automated registration</title>
|
|
<titleabbrev>Automated registration</titleabbrev>
|
|
|
|
<para role="first-line-indented">
|
|
To create a test case template registered in place of implementation, employ the macro
|
|
BOOST_AUTO_TEST_CASE_TEMPLATE. This facility is also called <firstterm>auto test case template</firstterm>.
|
|
</para>
|
|
|
|
<inline-synopsis>
|
|
<macro name="BOOST_AUTO_TEST_CASE_TEMPLATE" kind="functionlike">
|
|
<macro-parameter name="test_case_name"/>
|
|
<macro-parameter name="formal_type_parameter_name"/>
|
|
<macro-parameter name="collection_of_types"/>
|
|
</macro>
|
|
</inline-synopsis>
|
|
|
|
<para role="first-line-indented">
|
|
The macro BOOST_AUTO_TEST_CASE_TEMPLATE requires three arguments:
|
|
</para>
|
|
<variablelist>
|
|
<?dbhtml list-presentation="list"?>
|
|
<?dbhtml term-width="60%" list-width="100%"?>
|
|
<?dbhtml term-separator=" - "?> <!-- TO FIX: where separator? -->
|
|
|
|
<varlistentry>
|
|
<term>The test case template name</term>
|
|
<listitem>
|
|
<simpara>
|
|
unique test cases template identifier
|
|
</simpara>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>The name of a formal template parameter</term>
|
|
<listitem>
|
|
<simpara>
|
|
name of the type the test case template is instantiated with
|
|
</simpara>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>The collection of types to instantiate test case template with</term>
|
|
<listitem>
|
|
<simpara>
|
|
arbitrary MPL sequence
|
|
</simpara>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
<btl-example name="example10">
|
|
<title>Test case template with automated registration</title>
|
|
</btl-example>
|
|
</section>
|
|
|
|
<section id="utf.user-guide.test-organization.manual-test-case-template">
|
|
<title>Manually registered test case template</title>
|
|
<titleabbrev>Manual registration</titleabbrev>
|
|
|
|
<para role="first-line-indented">
|
|
One way to perform the same set of checks for a component instantiated with different template parameters is
|
|
illustrated in the following example:
|
|
</para>
|
|
|
|
<btl-snippet name="snippet2"/>
|
|
|
|
<para role="first-line-indented">
|
|
There several problems/inconveniencies with above approach, including:
|
|
<itemizedlist>
|
|
<listitem>
|
|
<simpara>
|
|
Fatal error in one of the invocation will stop whole test case and will skip invocations with different types
|
|
</simpara>
|
|
<simpara>
|
|
You need to repeat function invocation manually for all the parameters you are interested in
|
|
</simpara>
|
|
<simpara>
|
|
You need two functions to implement the test
|
|
</simpara>
|
|
</listitem>
|
|
</itemizedlist>
|
|
Ideally the test case template would be based on nullary function template (like single_test above).
|
|
Unfortunately function templates are neither addressable nor can be used as template parameters. To alleviate
|
|
the issue the manually registered test case template facility consists of two co-working macros:
|
|
BOOST_TEST_CASE_TEMPLATE_FUNCTION and BOOST_TEST_CASE_TEMPLATE. Former is used to define the test case
|
|
template body, later - to create and register test cases based on it.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
The macro BOOST_TEST_CASE_TEMPLATE_FUNCTION requires two arguments: the name of the test case template and the
|
|
name of the format type parameter:
|
|
</para>
|
|
|
|
<inline-synopsis>
|
|
<macro name="BOOST_TEST_CASE_TEMPLATE_FUNCTION" kind="functionlike">
|
|
<macro-parameter name="test_case_name"/>
|
|
<macro-parameter name="type_name"/>
|
|
</macro>
|
|
</inline-synopsis>
|
|
|
|
<btl-snippet name="snippet3"/>
|
|
|
|
<para role="first-line-indented">
|
|
The macro BOOST_TEST_CASE_TEMPLATE_FUNCTION is intended to be used in place of nullary function template
|
|
signature:
|
|
</para>
|
|
|
|
<btl-snippet name="snippet4"/>
|
|
|
|
<para role="first-line-indented">
|
|
The only difference is that the BOOST_TEST_CASE_TEMPLATE_FUNCTION makes the test case template name usable in
|
|
the template argument list.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
BOOST_TEST_CASE_TEMPLATE requires two arguments: the name of the test case template and Boost.MPL compatible
|
|
collection of types to instantiate it with. The names passed to both macros should be the same.
|
|
</para>
|
|
|
|
<inline-synopsis>
|
|
<macro name="BOOST_TEST_CASE_TEMPLATE" kind="functionlike">
|
|
<macro-parameter name="test_case_name"/>
|
|
<macro-parameter name="collection_of_types"/>
|
|
</macro>
|
|
</inline-synopsis>
|
|
|
|
<para role="first-line-indented">
|
|
BOOST_TEST_CASE_TEMPLATE creates an instance of the test case generator. When passed to the method
|
|
test_suite::add, the generator produces a separate sub test case for each type in the supplied collection of
|
|
types and registers it immediately in the test suite. Each test case is based on the test case template body
|
|
instantiated with a particular test type.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
Sub test case names are deduced from the macro argument test_case_name. If you prefer to assign different test
|
|
case names, you need to use the underlying make_test_case interface instead. Both test cases creation and
|
|
registration is performed in the test module initialization function.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
The test case template facility is preferable to the approach in example above, since execution of each sub
|
|
test case is guarded and counted separately. It produces a better test log/results report (in example above in
|
|
case of failure you can't say which type is at fault) and allows you to test all types even if one of
|
|
them causes termination of the sub test case.
|
|
</para>
|
|
|
|
<btl-example name="example09">
|
|
<title>Manually registered test case template</title>
|
|
</btl-example>
|
|
</section>
|
|
</section>
|
|
|
|
<section id="utf.user-guide.test-organization.test-suite">
|
|
<title>Test suite</title>
|
|
|
|
<para role="first-line-indented">
|
|
If you consider test cases as leaves on the test tree, the test suite can be considered as branch and the master
|
|
test suite as a trunk. Unlike real trees though, our tree in many cases consists only of leaves attached
|
|
directly to the trunk. This is common for all test cases to reside directly in the master test suite. If you do
|
|
want to construct a hierarchical test suite structure the &utf; provides both manual and automated
|
|
test suite creation and registration facilities:
|
|
</para>
|
|
|
|
<itemizedlist>
|
|
<listitem>
|
|
<simpara>
|
|
<link linkend="utf.user-guide.test-organization.auto-test-suite">Test suite with automated registration</link>
|
|
</simpara>
|
|
</listitem>
|
|
<listitem>
|
|
<simpara>
|
|
<link linkend="utf.user-guide.test-organization.manual-test-suite">Manually registered test suite</link>
|
|
</simpara>
|
|
</listitem>
|
|
</itemizedlist>
|
|
|
|
<para role="first-line-indented">
|
|
In addition the &utf; presents a notion of
|
|
<link linkend="utf.user-guide.test-organization.master-test-suite">Master Test Suite</link>. The most important
|
|
reason to learn about this component is that it provides an ability to access command line arguments supplied
|
|
to a test module.
|
|
</para>
|
|
|
|
<section id="utf.user-guide.test-organization.auto-test-suite">
|
|
<title>Test suites with automated registration</title>
|
|
<titleabbrev>Automated registration</titleabbrev>
|
|
|
|
<para role="first-line-indented">
|
|
The solution the &utf; presents for automated test suite creation and registration is designed to facilitate
|
|
multiple points of definition, arbitrary test suites depth and smooth integration with automated test case creation
|
|
and registration. This facility should significantly simplify a test tree construction process in comparison with
|
|
manual explicit registration case.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
The implementation is based on the order of file scope variables definitions within a single compilation unit.
|
|
The semantic of this facility is very similar to the namespace feature of C++, including support for test suite
|
|
extension. To start test suite use the macro BOOST_AUTO_TEST_SUITE. To end test suite use the macro
|
|
BOOST_AUTO_TEST_SUITE_END. The same test suite can be restarted multiple times inside the same test file or in a
|
|
different test files. In a result all test units will be part of the same test suite in a constructed test tree.
|
|
</para>
|
|
|
|
<inline-synopsis>
|
|
<macro name="BOOST_AUTO_TEST_SUITE" kind="functionlike">
|
|
<macro-parameter name="test_suite_name"/>
|
|
</macro>
|
|
<macro name="BOOST_AUTO_TEST_SUITE_END" kind="functionlike"/>
|
|
</inline-synopsis>
|
|
|
|
<para role="first-line-indented">
|
|
Test units defined in between test suite start and end declarations become members of the test suite. A test
|
|
unit always becomes the member of the closest test suite declared. Test units declared at a test file scope
|
|
become members of the master test suite. There is no limit on depth of test suite inclusion.
|
|
</para>
|
|
|
|
<btl-example name="example12">
|
|
<title>Test suites with automated registration</title>
|
|
|
|
<para role="first-line-indented">
|
|
This example creates a test tree that matches exactly the one created in the manual test suite registration
|
|
example. As you can see test tree construction in this example is more straightforward and automated.
|
|
</para>
|
|
</btl-example>
|
|
|
|
<btl-example name="example53">
|
|
<title>Example of test suite extension using automated registration facility</title>
|
|
|
|
<para role="first-line-indented">
|
|
In this example test suite test_suite consists of two parts. Their definition is remote and is separated by another
|
|
test case. In fact these parts may even reside in different test files. The resulting test tree remains the same. As
|
|
you can see from the output both test_case1 and test_case2 reside in the same test suite test_suite.
|
|
</para>
|
|
</btl-example>
|
|
|
|
</section>
|
|
|
|
<section id="utf.user-guide.test-organization.manual-test-suite">
|
|
<title>Manually registered test suites</title>
|
|
<titleabbrev>Manual registration</titleabbrev>
|
|
|
|
<para role="first-line-indented">
|
|
To create a test suite manually you need to create an instance of boost::unit_test::test_suite class, register
|
|
it in test tree and populate it with test cases (or lower level test suites).
|
|
</para>
|
|
|
|
<section id="utf.user-guide.test-organization.test-suite-registration-interface">
|
|
<title>Test unit registration interface</title>
|
|
|
|
<para role="first-line-indented">
|
|
The &utf; models the notion of test case container - test suite - using class boost::unit_test::test_suite. For
|
|
complete class interface reference check advanced section of this documentation. Here you should only be
|
|
interested in a single test unit registration interface:
|
|
</para>
|
|
|
|
<programlisting>void test_suite::add( test_unit* tc, counter_t expected_failures = 0, int timeout = 0 );</programlisting>
|
|
|
|
<para role="first-line-indented">
|
|
The first parameter is a pointer to a newly created test unit. The second optional parameter -
|
|
expected_failures - defines the number of test assertions that are expected to fail within the test unit. By
|
|
default no errors are expected.
|
|
</para>
|
|
|
|
<caution>
|
|
<simpara>
|
|
Be careful when supplying a number of expected failures for test suites. By default the &utf; calculates the
|
|
number of expected failures in test suite as the sum of appropriate values in all test units that constitute
|
|
it. And it rarely makes sense to change this.
|
|
</simpara>
|
|
</caution>
|
|
|
|
<para role="first-line-indented">
|
|
The third optional parameter - timeout - defines the timeout value for the test unit. As of now the &utf;
|
|
isn't able to set a timeout for the test suite execution, so this parameter makes sense only for test case
|
|
registration. By default no timeout is set. See the method
|
|
<methodname>boost::execution_monitor::execute</methodname> for more details about the timeout value.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
To register group of test units in one function call the boost::unit_test::test_suite provides another add
|
|
interface covered in the advanced section of this documentation.
|
|
</para>
|
|
</section>
|
|
|
|
<section id="utf.user-guide.test-organization.test-suite-instance-construction">
|
|
<title>Test suite instance construction</title>
|
|
|
|
<para role="first-line-indented">
|
|
To create a test suite instance manually, employ the macro BOOST_TEST_SUITE. It hides all implementation
|
|
details and you only required to specify the test suite name:
|
|
</para>
|
|
|
|
<inline-synopsis>
|
|
<macro name="BOOST_TEST_SUITE" kind="functionlike">
|
|
<macro-parameter name="test_suite_name"/>
|
|
</macro>
|
|
</inline-synopsis>
|
|
|
|
<para role="first-line-indented">
|
|
BOOST_TEST_SUITE creates an instance of the class boost::unit_test::test_suite and returns a pointer to the
|
|
constructed instance. Alternatively you can create an instance of class boost::unit_test::test_suite yourself.
|
|
</para>
|
|
|
|
<note>
|
|
<simpara>
|
|
boost::unit_test::test_suite instances have to be allocated on the heap and the compiler won't allow you
|
|
to create one on the stack.
|
|
</simpara>
|
|
</note>
|
|
|
|
<para role="first-line-indented">
|
|
Newly created test suite has to be registered in a parent one using add interface. Both test suite creation and
|
|
registration is performed in the test module initialization function.
|
|
</para>
|
|
</section>
|
|
|
|
<btl-example name="example11">
|
|
<title>Manually registered test suites</title>
|
|
</btl-example>
|
|
|
|
<para role="first-line-indented">
|
|
This example creates a test tree, which can be represented by the following hierarchy:
|
|
</para>
|
|
|
|
<mediaobject>
|
|
<imageobject>
|
|
<imagedata format="jpg" fileref="../img/class-hier.jpg"/>
|
|
</imageobject>
|
|
</mediaobject>
|
|
</section>
|
|
|
|
<section id="utf.user-guide.test-organization.master-test-suite">
|
|
<title>Master Test Suite</title>
|
|
|
|
<para role="first-line-indented">
|
|
As defined in introduction section the master test suite is a root node of a test tree. Each test module built
|
|
with the &utf; always has the master test suite defined. The &utf; maintain the master test suite instance
|
|
internally. All other test units are registered as direct or indirect children of the master test suite.
|
|
</para>
|
|
|
|
<programlisting>namespace boost {
|
|
namespace unit_test {
|
|
class master_test_suite_t : public test_suite {
|
|
public:
|
|
int argc;
|
|
char** argv;
|
|
};
|
|
|
|
} // namespace unit_test
|
|
|
|
} // namespace boost</programlisting>
|
|
|
|
<para role="first-line-indented">
|
|
To access single instance of the master test suite use the following interface:
|
|
</para>
|
|
|
|
<programlisting>namespace boost {
|
|
namespace unit_test {
|
|
namespace framework {
|
|
|
|
master_test_suite_t& master_test_suite();
|
|
|
|
} // namespace framework
|
|
} // namespace unit_test
|
|
} // namespace boost</programlisting>
|
|
|
|
<section id="utf.user-guide.test-organization.cla-access" >
|
|
<title>Command line arguments access interface</title>
|
|
|
|
<para role="first-line-indented">
|
|
Master test suite implemented as an extension to the regular test suite, since it maintains references to the
|
|
command line arguments passed to the test module. To access the command line arguments use
|
|
</para>
|
|
|
|
<programlisting>boost::unit_test::framework::master_test_suite().argc
|
|
boost::unit_test::framework::master_test_suite().argv</programlisting>
|
|
|
|
<para role="first-line-indented">
|
|
In below example references to the command line arguments are accessible either as an initialization function
|
|
parameters or as members of the master test suite. Both references point to the same values. A test module that
|
|
uses the alternative initialization function specification can only access command line arguments through the
|
|
master test suite.
|
|
</para>
|
|
|
|
<note>
|
|
<simpara>
|
|
This interface for runtime parameter access is temporary. It's planned to be updated once runtime
|
|
parameters support is redesigned.
|
|
</simpara>
|
|
</note>
|
|
|
|
<para role="first-line-indented">
|
|
Returning to the free function example, let's modify initialization function to check for absence of any
|
|
test module arguments.
|
|
</para>
|
|
|
|
<btl-example name="example13">
|
|
<title>Command line access in initialization function</title>
|
|
</btl-example>
|
|
</section>
|
|
|
|
<section id="utf.user-guide.test-organization.master-test-suite-name">
|
|
<title>Naming</title>
|
|
<para role="first-line-indented">
|
|
The master test suite is created with default name "Master Test Suite". There are two methods two
|
|
reset the name to a different value: using the macro <xref linkend="utf.flag.module" endterm="utf.flag.module"/>
|
|
and from within the test module initialization function. Former is used for test modules that don't have the
|
|
manually implemented initialization function. Following examples illustrate these methods.
|
|
</para>
|
|
|
|
<btl-example name="example14">
|
|
<title>Naming master test suite using the macro <xref linkend="utf.flag.module" endterm="utf.flag.module"/></title>
|
|
|
|
<para role="first-line-indented">
|
|
If the macro <xref linkend="utf.flag.module" endterm="utf.flag.module"/> is defined, the test module initialization
|
|
function is <link linkend="utf.user-guide.initialization.auto-generation">automatically generated</link> and the
|
|
macro value becomes the name of the master test suite. The name may include spaces.
|
|
</para>
|
|
</btl-example>
|
|
|
|
<btl-example name="example15">
|
|
<title>Naming master test suite explicitly in the test module initialization function</title>
|
|
|
|
<para role="first-line-indented">
|
|
Without the <xref linkend="utf.flag.main" endterm="utf.flag.main"/> and the <xref linkend="utf.flag.module"
|
|
endterm="utf.flag.module"/> flags defined, the test module initialization function has to be manually implemented.
|
|
The master test suite name can be reset at any point within this function.
|
|
</para>
|
|
</btl-example>
|
|
</section>
|
|
</section>
|
|
</section>
|
|
|
|
<section id="utf.user-guide.test-organization.expected-failures">
|
|
<title>Expected failures specification</title>
|
|
|
|
<para role="first-line-indented">
|
|
While in a perfect world all test assertions should pass in order for a test module to pass, in some situations
|
|
it is desirable to temporarily allow particular tests to fail. For example, where a particular feature is not
|
|
implemented yet and one needs to prepare a library for the release or when particular test fails on some
|
|
platforms. To avoid a nagging red box in regression tests table, you can use the expected failures feature.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
This feature allows specifying an expected number of failed assertions per test unit. The value is specified
|
|
during test tree construction, and can't be updated during test execution.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
The feature is not intended to be used to check for expected functionality failures. To check that a particular
|
|
input is causing an exception to be thrown use <macroname>BOOST_CHECK_THROW</macroname> family of testing
|
|
tools.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
The usage of this feature should be limited and employed only after careful consideration. In general you should
|
|
only use this feature when it is necessary to force a test module to pass without actually fixing the problem.
|
|
Obviously, an excessive usage of expected failures defeats the purpose of the unit test. In most cases it only
|
|
needs be applied temporarily.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
You also need to remember that the expected failure specification is per test case. This means that any failed
|
|
assertion within that test case can satisfy the expected failures quota. Meaning it is possible for an
|
|
unexpected failure to occur to satisfy this quota.
|
|
</para>
|
|
|
|
<note>
|
|
<simpara>
|
|
If an assertion at fault is fixed and passed, while an expected failures specification still present, the test
|
|
case is going to fail, since the number of failures is smaller than expected.
|
|
</simpara>
|
|
</note>
|
|
|
|
<section id="utf.user-guide.test-organization.manual-expected-failures">
|
|
<title>Usage with manually registered test cases</title>
|
|
|
|
<para role="first-line-indented">
|
|
To set the value of expected failures for the manually registered test unit pass it as a second argument for the
|
|
test_suite::add call during test unit registration.
|
|
</para>
|
|
|
|
<btl-example name="example16">
|
|
<title>Expected failures specification for manually registered test case</title>
|
|
</btl-example>
|
|
</section>
|
|
<section id="utf.user-guide.test-organization.auto-expected-failures">
|
|
<title>Usage with automatically registered test cases</title>
|
|
|
|
<para role="first-line-indented">
|
|
To set the number of expected failures for the automatically registered test unit use the macro
|
|
BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES before the test case definition.
|
|
</para>
|
|
|
|
<inline-synopsis>
|
|
<macro name="BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES" kind="functionlike">
|
|
<macro-parameter name="test_case_name"/>
|
|
<macro-parameter name="number_of_expected_failures"/>
|
|
</macro>
|
|
</inline-synopsis>
|
|
|
|
<para role="first-line-indented">
|
|
You can use this macro both on a file scope and inside a test suite. Moreover you can use it even if name of test
|
|
units coincide in different test suites. Expected failures specification applies to the test unit belonging to the same
|
|
test suite where BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES resides.
|
|
</para>
|
|
|
|
<btl-example name="example17">
|
|
<title>Expected failures specification for automatically registered test case</title>
|
|
</btl-example>
|
|
</section>
|
|
</section>
|
|
</section> |