test/doc/src/utf.user-guide.test-output.xml

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE section PUBLIC "-//Boost//DTD BoostBook XML V1.0//EN"  "../../../../tools/boostbook/dtd/boostbook.dtd" [
 <!ENTITY utf "<acronym>UTF</acronym>">
]>
<section id="utf.user-guide.test-output">
 <title>Test Output &hellip; or let's see what you got for your money</title>
 <titleabbrev>Test Output </titleabbrev>

 <para role="first-line-indented">
  The output produced by a test module is one of the major assets the &utf; brings to users. In comparison with any
  kind of manual/assert based solution the &utf; provide following services:
 </para>

 <itemizedlist>
  <listitem>
   <simpara>All test errors are reported uniformly</simpara>
   <simpara>
    The test execution monitor along with standardized output from all included
    <link linkend="utf.user-guide.testing-tools">testing tools</link> provides uniform reporting for all errors including fatal
    errors, like memory assess violation and uncaught exceptions.
   </simpara>
  </listitem>
  <listitem>
   <simpara>Detailed information on the source of an error</simpara>
   <simpara>
    The &utf; test tool's based assertion provides as much information as possible about cause of error,
    usually  allowing you to deduce what is wrong without entering the debugger or core analysis.
   </simpara>
  </listitem>
  <listitem>
   <simpara>
    Separation of the test errors description (test log) from the results report summary (test results report)
   </simpara>
   <simpara>
    The information produced during test execution, including all error, warning and info messages from the test
    tools, executed test units notification constitute the test log. By default all entries in the test log are
    directed to the standard output. Once testing is completed the &utf; may produce a summary test report with
    different levels of detail. The test report is by default directed to the standard error output.
   </simpara>
  </listitem>
  <listitem>
   <simpara>Flexibility in what is shown in the output</simpara>
   <simpara>
    The &utf; provides the ability to configure what if shown in both the test log and the test report. The
    configuration is supported both at runtime, during test module invocation and at compile time from within a
    test module.
   </simpara>
  </listitem>
  <listitem>
   <simpara>Flexibility in how output is formatted</simpara>
   <simpara>
    The &utf; provides the ability to configure the format of the test module output. At the moment only 2 formats
    are supported by the &utf; itself, the well defined public interface allows you to customize an output for
    your purposes almost any way you want.
   </simpara>
  </listitem>
 </itemizedlist>

 <section id="utf.user-guide.test-output.log">
  <title>Test log output</title>
  <titleabbrev>Test log</titleabbrev>

  <para role="first-line-indented">
   The test log is produced during the test execution. All entries in the test log are assigned a particular log
   level. Only the entries with level that exceeds the <firstterm>active log level threshold</firstterm> actually
   appear in the test log output. Log levels are arranged by the &quot;importance&quot; of the log entries. Here is
   the list of all levels in order of increasing &quot;importance&quot;:
  </para>

  <itemizedlist>
   <listitem>
    <simpara>Success information messages</simpara>
    <simpara>
     This category includes messages that provide information on successfully passed assertions
    </simpara>
   </listitem>
   <listitem>
    <simpara>Test tree traversal notifications</simpara>
    <simpara>
     This category includes messages that are produced by the &utf; core and indicate which test suites/cases are
     currently being executed or skipped
    </simpara>
   </listitem>
   <listitem>
    <simpara>General information messages</simpara>
    <simpara>
     This category includes general information massages produced in most cases by a test module author using the
     macro <macroname>BOOST_TEST_MESSAGE</macroname>.
    </simpara>
   </listitem>
   <listitem>
    <simpara>Warning messages</simpara>
    <simpara>
     This category includes messages produced by failed warning level assertions.
    </simpara>
   </listitem>
   <listitem>
    <simpara>Non fatal error messages</simpara>
    <simpara>
     This category includes messages produced by failed check level assertions
    </simpara>
   </listitem>
   <listitem>
    <simpara>Uncaught C++ exceptions notifications</simpara>
    <simpara>
     This category includes messages that are produced by the &utf; and provide detailed information on the C++
     exceptions uncaught by the test case body.
    </simpara>
   </listitem>
   <listitem>
    <simpara>Non-fatal system error</simpara>
    <simpara>
     This category includes messages that are produced by the &utf; itself and provides information about caught
     non-fatal system error. For example it includes messages produced in the case of test case timeout or if
     floating  point values calculation errors are caught.
    </simpara>
   </listitem>
   <listitem>
    <simpara>Fatal system error</simpara>
    <simpara>
     This category includes messages produced by failed require level assertions and by the &utf; itself in case of
     abnormal test case termination.
    </simpara>
   </listitem>
  </itemizedlist>

  <note>
   <simpara>
    The active log level works namely as threshold, not as selector. For the given active log level threshold, all
    test log entries with &quot;importance&quot; higher than threshold are enabled and all test log entries with
    &quot;importance&quot; below threshold are disabled.
   </simpara>
  </note>

  <para role="first-line-indented">
   In addition to the levels described above the test log defines two special log levels. The current log level can
   be set to:
  </para>

  <itemizedlist>
   <listitem>
    <simpara>All messages</simpara>
    <simpara>
     If active log level threshold is set to this value, all test log entries appear in the output. In practice
     this is equivalent to setting the active log level threshold to &quot;success information messages&quot;
    </simpara>
   </listitem>
   <listitem>
    <simpara>Nothing</simpara>
    <simpara>
     If the active log level threshold is set to this value, none of test log entries appear in the output. This log level
     is used to execute a &quot;silent&quot; test that doesn't produce any test log and only generates a result code indicating whether test failed or passed.
    </simpara>
   </listitem>
  </itemizedlist>

  <para role="first-line-indented">
   By default the active log level threshold is set to &quot;non fatal error messages&quot; and the test log output
   is generated in the human readable format. The active log level threshold and the output format can be configured
   at runtime during a test module invocation and at compile time from within a test module using the test log
   public interfaces. For example, for automated test module output processing it might be more convenient to use
   the XML based format.
  </para>

  <para role="first-line-indented">
   In most cases The &utf; can't provide an exact location, where system error occurs or uncaught C++ exception
   is thrown from. To be able to pinpoint it as close as possible the &utf; keeps track of checkpoints - the
   location a test module passed through. A test case entrance and exit points, a test tool invocation point the
   &utf; tracks automatically. Any other checkpoints should be entered by you manually. The test log provides two
   macros for this purpose: <macroname>BOOST_TEST_CHECKPOINT</macroname> - to specify a &quot;named&quot; checkpoint 
   and <macroname>BOOST_TEST_PASSPOINT</macroname> - to specify an &quot;unnamed&quot; checkpoint.
  </para>

  <section id="utf.user-guide.test-output.log.testing-tool-args">
   <title>Logging tool arguments</title>

   <para role="first-line-indented">
    Most of the <link linkend="utf.user-guide.testing-tools">testing tools</link> print values of their arguments to the output
    stream in some form of log statement. If arguments type does not support <code>operator&lt;&lt;(std::ostream&amp;,
    ArgumentType const&amp;)</code> interface you will get a compilation error. You can either implement above
    interface or prohibit the <link linkend="utf.user-guide.testing-tools">testing tools</link> from logging argument values for
    specified type. To do so use following statement on file level before first test case that includes statement
    failing to compile:
   </para>

   <inline-synopsis>
    <macro name="BOOST_TEST_DONT_PRINT_LOG_VALUE" kind="functionlike">
     <macro-parameter name="ArgumentType"/>
    </macro>
   </inline-synopsis>

   <btl-example name="example32">
    <title>BOOST_TEST_DONT_PRINT_LOG_VALUE usage</title>
    
    <simpara>
     Try to comment out BOOST_TEST_DONT_PRINT_LOG_VALUE statement and you end up with compile time error.
    </simpara>
   </btl-example>
  </section>

  <section id="utf.user-guide.test-output.log.runtime-config">
   <title>Runtime configuration</title>

   <para role="first-line-indented">
    The active log level threshold can be configured at runtime using the parameter
    <link linkend="utf.user-guide.runtime-config.reference">log_level</link>. The test log output format can be
    selected using either parameter <link linkend="utf.user-guide.runtime-config.reference">log_format</link> or the
    parameter <link linkend="utf.user-guide.runtime-config.reference">output_format</link>.
   </para>
  </section>

  <section id="utf.user-guide.test-output.log.BOOST_TEST_MESSAGE">
   <title>BOOST_TEST_MESSAGE</title>

   <para role="first-line-indented">
    The macro BOOST_TEST_MESSAGE is intended to be used for the purpose of injecting an additional message into the
    &utf; test log. These messages are not intended to indicate any error or warning conditions, but rather as
    information/status notifications. The macro signature is as follows:
   </para>

   <inline-synopsis>
    <macro name="BOOST_TEST_MESSAGE" kind="functionlike">
     <macro-parameter name="test_message"/>
    </macro>
   </inline-synopsis>

   <para role="first-line-indented">
    The test_message argument can be as simple as C string literal or any custom expression that you can produce
    with in a manner similar to standard iostream operation.
   </para>

   <important>
    <simpara>
     Messages generated by this tool do not appear in test log output with default value of the active log level
     threshold. For these messages to appear the active log level threshold has to be set to a value below or equal
     to &quot;message&quot;.
    </simpara>
   </important>

   <btl-example name="example21">
    <title>BOOST_TEST_MESSAGE usage</title>
   </btl-example>
  </section>

  <section id="utf.user-guide.test-output.log.BOOST_TEST_CHECKPOINT">
   <title>BOOST_TEST_CHECKPOINT</title>

   <para role="first-line-indented">
    The macro BOOST_TEST_CHECKPOINT is intended to be used to inject &quot;named&quot; checkpoint position. The
    macro signature is as follows:
   </para>

   <inline-synopsis>
    <macro name="BOOST_TEST_CHECKPOINT" kind="functionlike">
     <macro-parameter name="checkoint_message"/>
    </macro>
   </inline-synopsis>

   <para role="first-line-indented">
    The message formatted at the checkpoint position is saved and reported by the exception logging functions (if any
    occurs). Similarly to the <macroname>BOOST_TEST_MESSAGE</macroname> the message can be formatted from any standard
    output stream compliant components.
   </para>

   <btl-example name="example22">
    <title>BOOST_TEST_CHECKPOINT usage</title>
   </btl-example>
  </section>

  <section id="utf.user-guide.test-output.log.BOOST_TEST_PASSPOINT">
   <title>BOOST_TEST_PASSPOINT</title>

   <para role="first-line-indented">
    The macro BOOST_TEST_PASSPOINT is intended to be used to inject an &quot;unnamed&quot; checkpoint position. The
    macro signature is as follows:
   </para>

   <inline-synopsis>
    <macro name="BOOST_TEST_PASSPOINT" kind="functionlike">
    </macro>
   </inline-synopsis>
   
   <para role="first-line-indented">
    Unlike the macro <macroname>BOOST_TEST_CHECKPOINT</macroname> this macro doesn't require any message to be
    supplied with it. It's just a simple &quot;been there&quot; marker that records file name and line number
    code passes through.
   </para>

   <btl-example name="example23">
    <title>BOOST_TEST_PASSPOINT usage</title>
   </btl-example>
  </section>

  <section id="utf.user-guide.test-output.log.FPT">
   <title>Logging floating point type numbers</title>

   <para role="first-line-indented">
    It may appear that floating-point numbers are displayed by the &utf; with an excessive number of decimal digits.
    However the number of digits shown is chosen to avoid apparently nonsensical displays like <code>[1.00000 != 1.00000]</code>
    when comparing exactly unity against a value which is increased by just one least significant binary digit using
    the default precision for float of just 6 decimal digits, given by
    <classname>std::numeric_limits</classname>&lt;float&gt;::digits10. The function used for the number of decimal
    digits displayed is that proposed for a future C++ Standard,
    <ulink url="http://www2.open-std.org/JTC1/SC22/WG21/docs/papers/2005/n1822.pdf">A Proposal to add a max
    significant decimal digits value</ulink>, to be called <classname>std::numeric_limits</classname>::max_digits10();.
    For 32-bit floats, 9 decimal digits are needed to ensure a single bit change produces a different decimal digit
    string.
   </para>

   <para role="first-line-indented">
    So a much more helpful display using 9 decimal digits is thus:
    <computeroutput>[1.00000000 != 1.00000012]</computeroutput> showing that the two values are in fact different.
   </para>

   <para role="first-line-indented">
    For <acronym>IEEE754</acronym> 32-bit float values - 9 decimal digits are shown. For 64-bit <acronym>IEEE754</acronym> double - 17 decimal digits. For
    <acronym>IEEE754</acronym> extended long double using 80-bit - 21 decimal digits. For <acronym>IEEE754</acronym> quadruple long double 128-bit, and Sparc
    extended long double 128-bit - 36 decimal digits. For floating-point types, a convenient formula to calculate
    max_digits10 is: 2 + <classname>std::numeric_limits</classname>&lt;FPT&gt;::digits * 3010/10000;
   </para>

   <note>
    <simpara>
     Note that a user defined floating point type UDFPT must define
     <classname>std::numeric_limits</classname>&lt;UDFPT&gt;::is_specialized = true and provide an appropriate value
     for <classname>std::numeric_limits</classname>&lt;UDFPT&gt;::digits, the number of bits used for the significand
     or mantissa. For example, for the Sparc extended long double 128, 113 bits are used for the significand (one of
     which is implicit).
    </simpara>
   </note>
  </section>

  <section id="utf.user-guide.test-output.log.human-readabe-format">
   <title>Human readable log output format</title>
   <titleabbrev>Human readable format</titleabbrev>

   <para role="first-line-indented">
    The human readable log format is designed to closely match an errors description produced by the Microsoft family
    of C++ compilers. This format allows jumping to the error location, if test module output is redirected into IDE
    output window. The rest of the log messages are designed to produce the most human friendly description of the
    events occurring in test module. This is a default format generated by test modules.
   </para>

   <para role="first-line-indented">
    Here the list of events along with corresponding message and the condition that has to be satisfied for it to appear
    in the output.
   </para>

   <segmentedlist>
    <?dbhtml list-presentation="list"?>

    <segtitle>Event</segtitle>
    <segtitle>Condition</segtitle>
    <segtitle>Output</segtitle>

    <seglistitem>
     <seg>On testing start</seg>
     <seg>threshold != log_nothing</seg>
     <seg>
      <computeroutput><literallayout>Running <userinput>total number of test cases</userinput> test case(s) &hellip;</literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On testing start</seg>
     <seg>threshold != log_nothing and show_build_info is set</seg>
     <seg>
      <computeroutput><literallayout>Platform: $BOOST_PLATFORM
Compiler: $BOOST_COMPILER
STL     : $BOOST_STDLIB
Boost   : $BOOST_VERSION</literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On abnormal testing termination</seg>
     <seg>threshold &lt;= log_messages</seg>
     <seg>
      <computeroutput><literallayout>Test is aborted</literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On test unit start</seg>
     <seg>threshold &lt;= log_test_units</seg>
     <seg>
      <computeroutput><literallayout>Entering test <userinput>test unit type</userinput> <userinput>test unit name</userinput></literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On test unit end</seg>
     <seg>threshold &lt;= log_test_units; testing time is reported only if elapsed time is more than 1 mks.</seg>
     <seg>
      <computeroutput><literallayout>Leaving test <userinput>test unit type</userinput> <userinput>test unit name</userinput>; testing time <userinput>value</userinput></literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On skipped test unit</seg>
     <seg>threshold &lt;= log_test_units</seg>
     <seg>
      <computeroutput><literallayout>Test <userinput>test unit type</userinput> <userinput>test unit name</userinput> is skipped</literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On uncaught C++ exception</seg>
     <seg>threshold &lt;= log_cpp_exception_errors. Checkpoint message is reported only if provided</seg>
     <seg>
      <computeroutput><literallayout>unknown location(0): fatal error in <userinput>test case name</userinput>: <userinput>explanation</userinput>
<userinput>last checkpoint location</userinput>: last checkpoint: <userinput>checkpoint message</userinput></literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On resumable system error</seg>
     <seg>threshold &lt;= log_system_errors. Checkpoint message is reported only if provided</seg>
     <seg>
      <computeroutput><literallayout>unknown location(0): fatal error in <userinput>test case name</userinput>: <userinput>explanation</userinput>
<userinput>last checkpoint location</userinput>: last checkpoint: <userinput>checkpoint message</userinput></literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On fatal system error</seg>
     <seg>threshold &lt;= log_fatal_errors. Checkpoint message is reported only if provided</seg>
     <seg>
      <computeroutput><literallayout>unknown location(0): fatal error in <userinput>test case name</userinput>: <userinput>explanation</userinput>
<userinput>last checkpoint location</userinput>: last checkpoint: <userinput>checkpoint message</userinput></literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On passed test assertion</seg>
     <seg>threshold &lt;= log_successful_tests</seg>
     <seg>
      <computeroutput><literallayout><userinput>assertion location</userinput>: info: check<userinput>assertion expression</userinput> passed</literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On failed WARNING level test assertion</seg>
     <seg>threshold &lt;= log_warnings</seg>
     <seg>
      <computeroutput><literallayout><userinput>assertion location</userinput>: warning in <userinput>test case name</userinput>: condition <userinput>assertion description</userinput> is not satisfied</literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On failed CHECK level test assertion</seg>
     <seg>threshold &lt;= log_all_errors</seg>
     <seg>
      <computeroutput><literallayout><userinput>assertion location</userinput>: error in <userinput>test case name</userinput>: check <userinput>assertion description</userinput> failed</literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On failed REQUIRE level test assertion</seg>
     <seg>threshold &lt;= log_fatal_errors</seg>
     <seg>
      <computeroutput><literallayout><userinput>assertion location</userinput>: fatal error in <userinput>test case name</userinput>: critical check <userinput>assertion description</userinput> failed</literallayout></computeroutput>
     </seg>
    </seglistitem>

    <seglistitem>
     <seg>On test log message</seg>
     <seg>threshold &lt;= log_messages</seg>
     <seg>
      <computeroutput><literallayout><userinput>Message content</userinput></literallayout></computeroutput>
     </seg>
    </seglistitem>
   </segmentedlist>

   <para role="first-line-indented">
    Advanced <link linkend="utf.user-guide.testing-tools">testing tools</link> may produce more complicated error messages.
   </para>
  </section>

  <section id="utf.user-guide.test-output.log.xml-format">
   <title>XML based log output format</title>

   <para role="first-line-indented">
    This log format is designed for automated test results processing. The test log output XML schema depends on the
    active log level threshold.
   </para>

   <!-- TO FIX -->
  </section>
 
  <section id="utf.user-guide.test-output.log.ct-config">
   <title>Compile time configuration</title>
   <para role="first-line-indented">
    While many test log configuration tasks can be performed at runtime using predefined framework parameters, the 
    &utf; provides a compile time interface as well. The interface gives you full power over what, where and how to 
    log. The interface is provided by singleton class <classname>boost::unit_test::unit_test_log_t</classname> and is
    accessible through local file scope reference to single instance of this class: boost::unit_test::unit_test_log.
   </para>

   <section id="utf.user-guide.test-output.log.ct-config.output-stream">
    <title>Log output stream redirection</title>

    <para role="first-line-indented">
     If you want to redirect the test log output stream into something different from std::cout use the following 
     interface:
    </para>

    <programlisting>boost::unit_test::unit_test_log.set_stream( std::ostream&amp; str );</programlisting>

    <para role="first-line-indented">
     You can reset the output stream at any time both during the test module initialization and from within test 
     cases. There are no limitations on number of output stream resets either.
    </para>

    <btl-example name="example50">
     <title>Test log output redirection</title>

     <warning>
      <simpara>
       If you redirect test log output stream from global fixture setup, you are 
       <emphasis role="bold">required</emphasis> to reset it back to std::cout during teardown to prevent dangling
       references access
      </simpara>
     </warning>
    </btl-example>
   </section>

   <section id="utf.user-guide.test-output.log.ct-config.log-level">
    <title>Log level configuration</title>

    <para role="first-line-indented">
     If you need to enforce specific log level from within your test module use the following interface:
    </para>

    <programlisting>boost::unit_test::unit_test_log.set_threshold_level( boost::unit_test::log_level );</programlisting>

    <para role="first-line-indented">
     In regular circumstances you shouldn't use this interface, since you not only override default log level, but also
     the one supplied at test execution time. Prefer to use runtime parameters for log level selection.
    </para>

    <btl-example name="example51">
     <title>Compile time log level configuration</title>
    </btl-example>
   </section>

   <section id="utf.user-guide.test-output.log.ct-config.log-format">
    <title>Predefined log format selection</title>

    <para role="first-line-indented">
     The select at compile time the log format from the list of the formats supplied by the &utf; 
    </para>

    <programlisting>boost::unit_test::unit_test_log.set_format( boost::unit_test::output_format );</programlisting>

    <para role="first-line-indented">
     In regular circumstances you shouldn't use this interface. Prefer to use runtime parameters for predefined log 
     format selection.
    </para>

    <btl-example name="example52">
     <title>Compile time log format selection</title>
    </btl-example>
   </section>

   <section id="utf.user-guide.test-output.log.ct-config.log-formatter">
    <title>Custom log format support</title>
    
    <!-- TO FIX -->
   </section>
  </section>
 </section>

 <section id="utf.user-guide.test-output.results-report">
  <title>Test report output</title>
  <section id="utf.user-guide.test-output.results-report.runtime-config">
   <title>Runtime configuration</title>
   <!-- TO FIX -->
  </section>

  <section id="utf.user-guide.test-output.results-report.ct-config">
   <title>Compile time configuration</title>

   <section id="utf.user-guide.test-output.results-report.ct-config.output-stream">
    <title>Report output stream redirection and access</title>
    <!-- TO FIX -->
   </section>
   
   <section id="utf.user-guide.test-output.results-report.ct-config.report-level">
    <title>Report level configuration</title>
    <!-- TO FIX -->
   </section>

   <section id="utf.user-guide.test-output.results-report.ct-config.report-format">
    <title>Predefined report format selection</title>
   <!-- TO FIX -->
  </section>
   
   <section id="utf.user-guide.test-output.results-report.ct-config.report-formatter">
    <title>Custom report format support</title>
    <!-- TO FIX -->
   </section>

  </section>
 </section>

 <section id="utf.user-guide.test-output.progress">
  <title>Test progress display</title>
  <titleabbrev>Progress display</titleabbrev>
  <para role="first-line-indented">
   In case if the test module involves lengthy computation split among multiple test cases you may be interested in 
   progress monitor. The test runners supplied with the &utf; support simple text progress display, implemented based
   on <classname>boost::progress_display</classname><footnote>The &utf; interfaces allow implementing 
   an advanced GUI based test runner with arbitrary progress display controls</footnote>. The progress display output 
   is enabled using the &utf; parameter <link linkend="utf.user-guide.runtime-config.parameters">show_progress</link>.
  </para>    <!-- TO FIX: what's wrong with footnode? -->


  <para role="first-line-indented">
   The &utf; has no ability to estimate how long the test case execution is going to take and the manual test 
   progress update is not supported at this point. The &utf; tracks the progress on test case level. If you want to 
   see more frequent progress update, you need to split the test into multiple test cases.
  </para>

  <para role="first-line-indented">
   In default configuration both test log and test progress outputs are directed into standard output stream. Any test 
   log messages are going to interfere with test progress display. To prevent this you can either set log level to 
   lower level or redirect either test log or test progress output into different stream during test module 
   initialization. Use following interface to redirect test progress output:
  </para>

  <programlisting>boost::unit_test::progress_monitor.set_stream( std::ostream&amp; )</programlisting>

  <btl-example name="example49">
   <title>Progress report for the test module with large amount of test cases</title>
  </btl-example>
 </section>
</section>