test/doc/src/utf.user-guide.runtime-config.xml

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE section PUBLIC "-//Boost//DTD BoostBook XML V1.0//EN"  "../../../../tools/boostbook/dtd/boostbook.dtd" [
 <!ENTITY utf "<acronym>UTF</acronym>">
]>
<section id="utf.user-guide.runtime-config">
 <title>Runtime configuration &hellip; or what are the strings I can pull?</title>
 <titleabbrev>Runtime configuration </titleabbrev>

 <para role="first-line-indented">
  The &utf; supports multiple parameters that affect test module execution. To set the parameter's value you can
  either use a runtime configuration subsystem interface from within the test module initialization function or you can
  specify the value at runtime during test module invocation.
 </para>

 <para role="first-line-indented">
  The &utf; provides two ways to set a parameter at runtime: by specifying a command line argument and by setting an
  environment variable. The command line argument always overrides the corresponding environment variable.
 </para>

 <para role="first-line-indented">
  During test module initialization the &utf; parses the command line and excludes all parameters that belong to it and
  their values from the argument list. The rest of command line is forwarded to the test module initialization function
  supplied by you. The command line argument format expected by the &utf; is:
 </para>

 <simpara> <!-- TO FIX -->
  --&lt;command line argument name&gt;=&lt;argument_value&gt;.
 </simpara>

 <para role="first-line-indented">
  The command line argument name is case sensitive. It is required to match exactly the name in parameter specification.
  There should not be any spaces between '=' and either command line argument name or argument value.
 </para>

 <para role="first-line-indented">
  The corresponding environment variable name is also case sensitive and is required to exactly match the name in the
  parameter specification.
 </para>

 <para role="first-line-indented">
  All information about supported parameters is summarized below in the reference section.
 </para>

 <section id="utf.user-guide.runtime-config.run-by-name">
  <title>Running specific test units selected by their name</title>
  <titleabbrev>Run by name</titleabbrev>

  <para role="first-line-indented">
   In regular circumstances test module execution initiates testing of all test units manually or automatically 
   registered in master test suite. The &utf; provides an ability to run specific set of test unit as well. It can be 
   single test case, single test suite or some combination of test cases and suites. The tests units to run are 
   selected by the runtime parameter <link linkend="utf.user-guide.runtime-config.reference">run_test</link>. In the 
   following examples I select tests to run using command line arguments, but the same filter expression can be used as 
   an appropriate environment variable value.
  </para>

  <para role="first-line-indented">
   Filter expressions are specified in a form a/b/c, where a, b and c are filters for corresponding levels of test tree. 
   Symbol '*' can be used at the beginning, at the end and as the level filter itself as an asterisk. Symbol ',' is used
   to create list of test units residing on the same level in test tree. 
  </para>

  <para role="first-line-indented">
   Let's consider following test module consisting from several test suites and test cases.
  </para>
  
  <btl-snippet name="snippet18"/>

  <itemizedlist>
   <listitem>
    <simpara>Run single test case by specifying it's name.</simpara>

    <screen html-class="test-execution-output">&gt;example --log_level=test_suite --run_test=testA
Running 1 test case...
Entering test suite "example"
Entering test case "testA"
Test case testA doesn't include any assertions
Leaving test case "testA"
Leaving test suite "example"

*** No errors detected</screen>
   </listitem>
   <listitem>
    <simpara>
     Running multiple test cases residing within the same test suite by listing their names in coma separated list.
    </simpara>

    <screen html-class="test-execution-output">&gt;example --log_level=test_suite --run_test=testA,testB
Running 2 test case...
Entering test suite "example"
Entering test case "testA"
Test case testA doesn't include any assertions
Leaving test case "testA"
Entering test case "testB"
Test case testA doesn't include any assertions
Leaving test case "testB"
Leaving test suite "example"

*** No errors detected</screen>
   </listitem>
   <listitem>
    <simpara>Incorrect test case name may lead to no test to be run.</simpara>

    <screen html-class="test-execution-output">&gt;example --log_level=test_suite --run_test=testC
Test setup error: no test cases matching filter</screen>
   </listitem>
   <listitem>
    <simpara>
     Test unit name can refer to a test suite as well. All test units within the referred test suites are going to be 
     run.
    </simpara>

    <screen html-class="test-execution-output">&gt;example --log_level=test_suite --run_test=s1
Running 2 test cases...
Entering test suite "example"
Entering test suite "s1"
Entering test case "test1"
Test case test1 doesn't include any assertions
Leaving test case "test1"
Entering test case "lest2"
Test case lest2 doesn't include any assertions
Leaving test case "lest2"
Leaving test suite "s1"
Leaving test suite "example"

*** No errors detected</screen>
   </listitem>
   <listitem>
    <simpara>
     Using '/' as levels separator you can refer to any test unit inside a test tree.
    </simpara>

    <screen html-class="test-execution-output">&gt;example --log_level=test_suite --run_test=s2/in/test
Running 1 test case...
Entering test suite "example"
Entering test suite "s2"
Entering test suite "in"
Entering test case "test"
Test case test doesn't include any assertions
Leaving test case "test"
Leaving test suite "in"
Leaving test suite "s2"
Leaving test suite "example"

*** No errors detected</screen>
   </listitem>
   <listitem>
    <simpara>
     The &utf; supports simple regular expression-like '*' wildcard. Single '*' match any name of test unit. Accordingly
     expression 's1/*' is equivalent to the expression 's1' and matches all test units inside test suite s1. Similarly 
     expression '*/test1' refers to all test units named test1 that reside in master test suite's direct child suites.
    </simpara>

    <screen html-class="test-execution-output">&gt;example --log_level=test_suite --run_test=*/test1
Running 2 test cases...
Entering test suite "example"
Entering test suite "s1"
Entering test case "test1"
Test case test1 doesn't include any assertions
Leaving test case "test1"
Leaving test suite "s1"
Entering test suite "s2"
Entering test case "test1"
Test case test1 doesn't include any assertions
Leaving test case "test1"
Leaving test suite "s2"
Leaving test suite "example"

*** No errors detected</screen>
   </listitem>
   <listitem>
    <simpara>
     The &utf; allows to match specific prefix in test unit names. For example, expression 's2/test*' filters out only 
     test units in test suite s2 with name that starts with 'test'. This avoids running test suite s2/in.
    </simpara>

    <screen html-class="test-execution-output">&gt;example --log_level=test_suite --run_test=s2/test*
Running 2 test cases...
Entering test suite "example"
Entering test suite "s2"
Entering test case "test1"
Test case test1 doesn't include any assertions
Leaving test case "test1"
Entering test case "test11"
Test case test11 doesn't include any assertions
Leaving test case "test11"
Leaving test suite "s2"
Leaving test suite "example"

*** No errors detected</screen>
   </listitem>
   <listitem>
    <simpara>
     The &utf; allows to match specific suffix in test unit names. For example, expression '*/*1' filters out test units
     with name that ends with '1' and reside in master test suite's direct child suites.
    </simpara>

    <screen html-class="test-execution-output">&gt;example --log_level=test_suite --run_test=*/*1
Running 2 test cases...
Entering test suite "example"
Entering test suite "s2"
Entering test case "test1"
Test case test1 doesn't include any assertions
Leaving test case "test1"
Entering test case "test11"
Test case test11 doesn't include any assertions
Leaving test case "test11"
Leaving test suite "s2"
Leaving test suite "example"

*** No errors detected</screen>
   </listitem>
   <listitem>
    <simpara>
     Finally, the &utf; allows to match specific substring in test unit names.
    </simpara>

    <screen html-class="test-execution-output">&gt;example --log_level=test_suite --run_test=s1/*est*
Running 2 test cases...
Entering test suite "example"
Entering test suite "s1"
Entering test case "test1"
Test case test1 doesn't include any assertions
Leaving test case "test1"
Entering test case "lest2"
Test case lest2 doesn't include any assertions
Leaving test case "lest2"
Leaving test suite "s1"
Leaving test suite "example"

*** No errors detected</screen>
   </listitem>
  </itemizedlist>
 </section>
 
 <section id="utf.user-guide.runtime-config.reference">
  <title>Runtime parameters reference</title>
  <titleabbrev>Parameters reference</titleabbrev>

  <para role="first-line-indented">
   Each parameter specification includes: the full parameter name, corresponding environment variable name, command
   line argument name, acceptable values and a long description. The default value for the parameter is bold in the
   acceptable values list. All values are case sensitive and are required to exactly match the parameter specification.
  </para>

  <btl-parameter-reference id="utf.user-guide.runtime-config.parameters">
   <refentry name="auto_start_dbg">
    <name>Automatically attach debugger in case of system failure</name>
    <env>BOOST_TEST_AUTO_START_DBG</env>
    <cla>auto_start_dbg</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">no</emphasis></member>
      <member>yes</member>
      <member>debugger identifier</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      specifies whether Boost.Test should try to attach a debugger in case if fatal system error occurs. If value is "yes"
      the default debugger configured for the platform is going to be attempted. Alternatively the debugger identified 
      by the argument value of the parameter is used. For more details on advanced debugger support in Boost.Test check 
      <!-- TO FIX: add link --> section dedicated to Boost.Test debug API.
     </simpara>
    </descr>
   </refentry>

   <refentry name="break_exec_path">
    <name>Break execution path</name>
    <env>BOOST_TEST_BREAK_EXEC_PATH</env>
    <cla>break_exec_path"</cla>
    <vals>
     <simplelist>
      <member>string consisting of space separate test_name:execution_path_number pairs</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      this runtime parameter is used by exception safety tester. By default exception safety tester only reports index of 
      execution path and  test case name where failure occurred. Using this parameter you can make the tester to break the 
      execution right before entering this path.
     </simpara>
    </descr>
   </refentry>

   <refentry name="build_info">
    <name>Print build info</name>
    <env>BOOST_TEST_BUILD_INFO</env>
    <cla>build_info</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">no</emphasis></member>
      <member>yes</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      makes the framework to print build information that include: platform, compiler, STL implementation in use and
      boost version.
     </simpara>
    </descr>
   </refentry>

   <refentry name="catch_system_errors">
    <name>Catch system errors</name>
    <env>BOOST_TEST_CATCH_SYSTEM_ERRORS</env>
    <cla>catch_system_errors</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">yes</emphasis></member>
      <member>no</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      value "no" prohibits the framework from catching asynchronous system events. This could be used for test programs
      executed within GUI or to get a coredump for stack analysis. See usage recommendations page for more details.
     </simpara>
    </descr>
   </refentry>

   <refentry name="color_output">
    <name>Produce color output</name>
    <env>BOOST_TEST_COLOR_OUTPUT</env>
    <cla>color_output</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">no</emphasis></member>
      <member>yes</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      The &utf; is able to produce color output on systems which supports it. To enable this behavior set the parameter to 
      'yes'. By default the output produces in not colored.
     </simpara>
    </descr>
   </refentry>

   <refentry name="detect_fp_exceptions">
    <name>[Do not] trap floating point exceptions</name>
    <env>BOOST_TEST_DETECT_FP_EXCEPTIONS</env>
    <cla>detect_fp_exceptions</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">no</emphasis></member>
      <member>yes</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      enables/disable hardware traps for the floating point exception if they are supported. By default is disabled.
     </simpara>
    </descr>
   </refentry>

   <refentry name="detect_memory_leaks">
    <name>Detect memory leaks</name>
    <env>BOOST_TEST_DETECT_MEMORY_LEAK</env>
    <cla>detect_memory_leaks</cla>
    <vals>
     <simplelist>
      <member>0</member>
      <member><emphasis role="bold">1</emphasis></member>
      <member>integer value &gt; 1</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      Positive value tells the framework to detect the memory leaks (if any). In addition any value greater than 1
      is treated as leak allocation number and setups runtime breakpoint. In other words setting this parameter to
      the positive value N greater than 1 causes the framework to set a breakpoint at Nth memory allocation (don't
      do that from the command line - only when you are under debugger). Note: if your test program produces memory
      leaks notifications, they are combined with allocation number values you could use to set a breakpoint.
      Currently only applies to MS family of compilers in debug builds.
     </simpara>
    </descr>
   </refentry>

   <refentry name="log_format">
    <name>The log format</name>
    <env>BOOST_TEST_LOG_FORMAT</env>
    <cla>log_format</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">HRF</emphasis> - human readable format</member>
      <member>XML - XML format for automated output processing</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      allows selecting the &utf; log format from the list of formats supplied by the framework. To specify custom log
      format use the <link linkend="utf.user-guide.test-output.log.ct-config.log-formatter">unit_test_log API</link>.
     </simpara>
    </descr>
   </refentry>

   <refentry name="log_level">
    <name>The &utf; log level</name>
    <env>BOOST_TEST_LOG_LEVEL</env>
    <cla>log_level</cla>
    <vals>
     <variablelist>
      <?dbhtml term-separator=" - "?>

      <varlistentry>
       <term>all</term>
       <listitem><simpara>report all log messages including the passed test notification</simpara></listitem>
      </varlistentry>
      <varlistentry>
       <term>success</term>
       <listitem><simpara>the same as all</simpara></listitem>
      </varlistentry>
      <varlistentry>
       <term>test_suite</term>
       <listitem><simpara>show test suite messages</simpara></listitem>
      </varlistentry>
      <varlistentry>
       <term>message</term>
       <listitem><simpara>show user messages</simpara></listitem>
      </varlistentry>
      <varlistentry>
       <term>warning</term>
       <listitem><simpara>report warnings issued by user</simpara></listitem>
      </varlistentry>
      <varlistentry>
       <term><emphasis role="bold">error</emphasis></term>
       <listitem><simpara>report all error conditions</simpara></listitem>
      </varlistentry>
      <varlistentry>
       <term>cpp_exception</term>
       <listitem><simpara>report uncaught c++ exception</simpara></listitem>
      </varlistentry>
      <varlistentry>
       <term>system_error</term>
       <listitem><simpara>report system originated non-fatal errors (for example, timeout or floating point exception)</simpara></listitem>
      </varlistentry>
      <varlistentry>
       <term>fatal_error</term>
       <listitem><simpara>report only user or system originated fatal errors (for example, memory access violation)</simpara></listitem>
      </varlistentry>
      <varlistentry>
       <term>nothing</term>
       <listitem><simpara>do not report any information</simpara></listitem>
      </varlistentry>
     </variablelist>
    </vals>
    <descr>
     <simpara>
      allows setting the &utf; <link linkid="utf.user-guide.test-output.log">log level</link> in a range from a
      complete log, when all successful tests are confirmed and  all test suite messages are included, to an empty
      log when nothing is logged a test output stream.  Note that log levels are accumulating, in other words each
      log level includes also all the information reported  by less  restrictive ones.
     </simpara>
    </descr>
   </refentry>

   <refentry name="log_sink">
    <name>The log sink name</name>
    <env>BOOST_TEST_LOG_SINK</env>
    <cla>log_sink</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">stdout</emphasis></member>
      <member>stderr</member>
      <member>arbitrary file name</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      This parameter allows easily redirect the test log. The parameter value is the string containing either a file 
      name, in which case the &utf; will redirect log into file with that name, or 'stdout', in which case log is
      redirected into standard output stream, or 'stderr' , in which case log is redirected into standard error stream.
      Default is 'stdout'
     </simpara>
    </descr>
   </refentry>

   <refentry name="output_format">
    <name>The output format</name>
    <env>BOOST_TEST_OUTPUT_FORMAT</env>
    <cla>output_format</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">HRF</emphasis> - human readable format</member>
      <member>XML - XML format for automated output processing</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      combines an effect of report_format and log_format parameters. Has higher priority than either one of them if
      specified.
     </simpara>
    </descr>
   </refentry>

   <refentry name="random">
    <name>Random seed for random order of test cases</name>
    <env>BOOST_TEST_RANDOM</env>
    <cla>random</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">0</emphasis></member>
      <member>1</member>
      <member>integer value &gt; 1</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      positive value makes the framework to run the test cases in random order. Also if this value is greater than 1,
      it's used as a random seed. In other case random seed is generated based on current time.
     </simpara>
    </descr>
   </refentry>

   <refentry name="report_format">
    <name>The report format</name>
    <env>BOOST_TEST_REPORT_FORMAT</env>
    <cla>report_format</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">HRF</emphasis> - human readable format</member>
      <member>XML - XML format for automated output processing</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      allows selecting the &utf; report format from the list of formats supplied by the framework. To
      specify custom report format use unit_test_report API.
     </simpara>
    </descr>
   </refentry>

   <refentry name="report_level">
    <name>The results report level</name>
    <env>BOOST_TEST_REPORT_LEVEL</env>
    <cla>report_level</cla>
    <vals>
     <simplelist>
      <member>no</member>
      <member><emphasis role="bold">confirm</emphasis></member>
      <member>short</member>
      <member>detailed</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      allows setting the level of detailization for the testing results report generated by the framework. Use value 
      "no" to eliminate the results report completely. See the 
      <xref linkend="utf.user-guide.test-output.results-report"/> for description of different report formats.
     </simpara>
    </descr>
   </refentry>

   <refentry name="report_sink">
    <name>The report sink name</name>
    <env>BOOST_TEST_REPORT_SINK</env>
    <cla>report_sink</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">stderr</emphasis></member>
      <member>stdout</member>
      <member>arbitrary file name</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      This parameter allows easily redirect the test results report. The parameter value is the string containing either 
      a file name, in which case the &utf; will redirect results report into file with that name, or 'stdout', in which case 
      report is redirected into standard output stream, or 'stderr', in which case report is redirected into standard error 
      stream. Default is 'stderr'.
     </simpara>
    </descr>
   </refentry>

   <refentry name="result_code">
    <name>[Do not] return result code</name>
    <env>BOOST_TEST_RESULT_CODE</env>
    <cla>result_code</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">yes</emphasis></member>
      <member>no</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      value "no" enforces the framework to always return zero result code. This could be used for test programs
      executed within GUI. See <link linkend="utf.usage-recommendations.dot-net-specific">usage recommendations</link>
      section for more details.
     </simpara>
    </descr>
   </refentry>
   
   <refentry name="run_test">
    <name>Test units to run</name>
    <env>BOOST_TESTS_TO_RUN</env>
    <cla>run_test</cla>
    <vals>
     <simplelist>
      <member>
       <link linkend="utf.user-guide.runtime-config.run-by-name">specification</link> of test units to run
      </member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      specifies which test units to run.
     </simpara>
    </descr>
   </refentry>

   <refentry name="save_patterm">
    <name>Save patterm</name>
    <env>BOOST_TEST_SAVE_PATTERN</env>
    <cla>save_pattern</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">no</emphasis></member>
      <member>yes</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      this parameter serves no particular purpose within the framework itself. It can be used by test modules relying 
      on output_test_stream to implement testing logic. output_test_stream has two modes of operation: save the pattern 
      file and match against stored pattern. You can use this parameter to switch between these modes, by passing the 
      parameter value to the output_test_stream constructor.
     </simpara>
    </descr>
   </refentry>

   <refentry name="show_progress">
    <name>Show progress</name>
    <env>BOOST_TEST_SHOW_PROGRESS</env>
    <cla>show_progress</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">no</emphasis></member>
      <member>yes</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>makes the framework to print progress information.</simpara>
    </descr>
   </refentry>
   
   <refentry name="use_alt_stack">
    <name>Use alternative stack</name>
    <env>BOOST_TEST_USE_ALT_STACK</env>
    <cla>use_alt_stack</cla>
    <vals>
     <simplelist>
      <member><emphasis role="bold">yes</emphasis></member>
      <member>no</member>
     </simplelist>
    </vals>
    <descr>
     <simpara>
      specifies whether or not the Boost.Test Execution Monitor should employ an alternative stack for signals 
      processing on platforms where they are supported.
     </simpara>
    </descr>
   </refentry>

  </btl-parameter-reference>
 </section>
</section>