mirror of
https://github.com/boostorg/test.git
synced 2026-02-18 14:32:11 +00:00
191 lines
9.7 KiB
HTML
Executable File
191 lines
9.7 KiB
HTML
Executable File
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
|
|
<title>Expected failures specification</title>
|
|
<link rel="stylesheet" href="../../../../style/style.css" type="text/css">
|
|
<meta name="generator" content="DocBook XSL Stylesheets V1.76.1">
|
|
<link rel="home" href="../../../index.html" title="Boost Test Library">
|
|
<link rel="up" href="../test-organization.html" title="Test organization … or the house that Jack built">
|
|
<link rel="prev" href="master-test-suite.html" title="Master Test Suite">
|
|
<link rel="next" href="../testing-tools.html" title="The UTF testing tools … or tester's toolbox for all occasions">
|
|
<script language="JavaScript1.2" src="../../../../js/boost-test.js"></script>
|
|
</head>
|
|
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
|
|
<table width="100%"><tr>
|
|
<td width="10%"><a href="../../../index.html"><img alt="Home" width="229" height="61" border="0" src="../../../../../../../libs/test/doc/img/boost.test.logo.png"></a></td>
|
|
<td valign="middle" align="left"> > <a href="../../../utf.html">The Unit Test Framework</a><a href="../../../execution-monitor.html">
|
|
>
|
|
</a><a href="../../user-guide.html">User's guide</a><a href="../../usage-recommendations.html">
|
|
>
|
|
</a><a href="../test-organization.html">Test organization</a><a href="../testing-tools.html">
|
|
>
|
|
</a><b>Expected failures specification</b>
|
|
</td>
|
|
<td><div class="spirit-nav">
|
|
<a href="master-test-suite.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a href="../testing-tools.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
|
|
</div></td>
|
|
</tr></table>
|
|
<hr>
|
|
<div class="section">
|
|
<div class="titlepage"><div><div><h5 class="title">
|
|
<a name="utf.user-guide.test-organization.expected-failures"></a>Expected failures specification</h5></div></div></div>
|
|
<p class="first-line-indented">
|
|
While in a perfect world all test assertions should pass in order for a test module to pass, in some situations
|
|
it is desirable to temporarily allow particular tests to fail. For example, where a particular feature is not
|
|
implemented yet and one needs to prepare a library for the release or when particular test fails on some
|
|
platforms. To avoid a nagging red box in regression tests table, you can use the expected failures feature.
|
|
</p>
|
|
<p class="first-line-indented">
|
|
This feature allows specifying an expected number of failed assertions per test unit. The value is specified
|
|
during test tree construction, and can't be updated during test execution.
|
|
</p>
|
|
<p class="first-line-indented">
|
|
The feature is not intended to be used to check for expected functionality failures. To check that a particular
|
|
input is causing an exception to be thrown use <code class="computeroutput"><a class="link" href="../testing-tools/reference.html" title="The UTF testing tools reference">BOOST_CHECK_THROW</a></code> family of testing
|
|
tools.
|
|
</p>
|
|
<p class="first-line-indented">
|
|
The usage of this feature should be limited and employed only after careful consideration. In general you should
|
|
only use this feature when it is necessary to force a test module to pass without actually fixing the problem.
|
|
Obviously, an excessive usage of expected failures defeats the purpose of the unit test. In most cases it only
|
|
needs be applied temporarily.
|
|
</p>
|
|
<p class="first-line-indented">
|
|
You also need to remember that the expected failure specification is per test case. This means that any failed
|
|
assertion within that test case can satisfy the expected failures quota. Meaning it is possible for an
|
|
unexpected failure to occur to satisfy this quota.
|
|
</p>
|
|
<div class="note"><table border="0" summary="Note">
|
|
<tr>
|
|
<td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../../../../doc/src/images/note.png"></td>
|
|
<th align="left">Note</th>
|
|
</tr>
|
|
<tr><td align="left" valign="top"><p>
|
|
If an assertion at fault is fixed and passed, while an expected failures specification still present, the test
|
|
case is going to fail, since the number of failures is smaller than expected.
|
|
</p></td></tr>
|
|
</table></div>
|
|
<div class="section">
|
|
<div class="titlepage"><div><div><h6 class="title">
|
|
<a name="utf.user-guide.test-organization.manual-expected-failures"></a>Usage with manually registered test cases</h6></div></div></div>
|
|
<p class="first-line-indented">
|
|
To set the value of expected failures for the manually registered test unit pass it as a second argument for the
|
|
test_suite::add call during test unit registration.
|
|
</p>
|
|
<div class="example">
|
|
<a name="utf.user-guide.test-organization.manual-expected-failures.example16"></a><p class="title"><b>Example 17. Expected failures specification for manually registered test case</b></p>
|
|
<div class="example-contents">
|
|
<pre class="programlisting">#include <boost/test/included/unit_test.hpp>
|
|
using namespace boost::unit_test;
|
|
|
|
//____________________________________________________________________________//
|
|
|
|
void free_test_function()
|
|
{
|
|
BOOST_CHECK( 2 == 1 );
|
|
}
|
|
|
|
//____________________________________________________________________________//
|
|
|
|
test_suite*
|
|
init_unit_test_suite( int, char* [] ) {
|
|
framework::master_test_suite().
|
|
add( BOOST_TEST_CASE( &free_test_function ), 2 );
|
|
|
|
return 0;
|
|
}
|
|
|
|
//____________________________________________________________________________//
|
|
</pre>
|
|
<table border="0" summary="Simple list" class="simplelist"><tr>
|
|
<td><code class="literal"><a href="../../../../src/examples/example16.cpp" target="_top">Source code</a></code></td>
|
|
<td> | </td>
|
|
<td><code class="literal"><a href="#" target="_top" id="id588251" onclick="toggle_element( 'example16-output', 'id588251', 'Show output', 'Hide output' ); return false;">Show output</a></code></td>
|
|
</tr></table>
|
|
<pre class="example-output" id="example16-output">> example --log_level=message
|
|
Running 1 test case...
|
|
test.cpp(8): error in "free_test_function": check 2 == 1 failed
|
|
Test case has less failures then expected
|
|
|
|
*** No errors detected
|
|
</pre>
|
|
</div>
|
|
</div>
|
|
<br class="example-break">
|
|
</div>
|
|
<div class="section">
|
|
<div class="titlepage"><div><div><h6 class="title">
|
|
<a name="utf.user-guide.test-organization.auto-expected-failures"></a>Usage with automatically registered test cases</h6></div></div></div>
|
|
<p class="first-line-indented">
|
|
To set the number of expected failures for the automatically registered test unit use the macro
|
|
BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES before the test case definition.
|
|
</p>
|
|
<pre class="inline-synopsis">
|
|
<a name="BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES"></a>BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES(<span class="emphasis"><em>test_case_name</em></span>, <span class="emphasis"><em>number_of_expected_failures</em></span>)</pre>
|
|
<p class="first-line-indented">
|
|
You can use this macro both on a file scope and inside a test suite. Moreover you can use it even if name of test
|
|
units coincide in different test suites. Expected failures specification applies to the test unit belonging to the same
|
|
test suite where BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES resides.
|
|
</p>
|
|
<div class="example">
|
|
<a name="utf.user-guide.test-organization.auto-expected-failures.example17"></a><p class="title"><b>Example 18. Expected failures specification for automatically registered test case</b></p>
|
|
<div class="example-contents">
|
|
<pre class="programlisting">#define BOOST_TEST_MODULE example
|
|
#include <boost/test/included/unit_test.hpp>
|
|
|
|
//____________________________________________________________________________//
|
|
|
|
BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( my_test1, 1 )
|
|
|
|
BOOST_AUTO_TEST_CASE( my_test1 )
|
|
{
|
|
BOOST_CHECK( 2 == 1 );
|
|
}
|
|
|
|
//____________________________________________________________________________//
|
|
|
|
BOOST_AUTO_TEST_SUITE( internal )
|
|
|
|
BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( my_test1, 2 )
|
|
|
|
BOOST_AUTO_TEST_CASE( my_test1 )
|
|
{
|
|
BOOST_CHECK_EQUAL( sizeof(int), sizeof(char) );
|
|
BOOST_CHECK_EQUAL( sizeof(int*), sizeof(char) );
|
|
}
|
|
|
|
//____________________________________________________________________________//
|
|
|
|
BOOST_AUTO_TEST_SUITE_END()</pre>
|
|
<table border="0" summary="Simple list" class="simplelist"><tr>
|
|
<td><code class="literal"><a href="../../../../src/examples/example17.cpp" target="_top">Source code</a></code></td>
|
|
<td> | </td>
|
|
<td><code class="literal"><a href="#" target="_top" id="id588338" onclick="toggle_element( 'example17-output', 'id588338', 'Show output', 'Hide output' ); return false;">Show output</a></code></td>
|
|
</tr></table>
|
|
<pre class="example-output" id="example17-output">> example --report_level=short
|
|
Running 2 test cases...
|
|
test.cpp(10): error in "my_test1": check 2 == 1 failed
|
|
test.cpp(21): error in "my_test1": check sizeof(int) == sizeof(char) failed [4 != 1]
|
|
test.cpp(22): error in "my_test1": check sizeof(int*) == sizeof(char) failed [4 != 1]
|
|
|
|
Test suite "example" passed with:
|
|
3 assertions out of 3 failed
|
|
3 failures expected
|
|
2 test cases out of 2 passed
|
|
</pre>
|
|
</div>
|
|
</div>
|
|
<br class="example-break">
|
|
</div>
|
|
</div>
|
|
<table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr>
|
|
<td align="left"></td>
|
|
<td align="right"><div class="copyright-footer">Copyright © 2001-2011 Gennadiy Rozental</div></td>
|
|
</tr></table>
|
|
<hr>
|
|
<div class="spirit-nav">
|
|
<a accesskey="p" href="master-test-suite.html"><img src="../../../../../../../doc/src/images/prev.png" alt="Prev"></a><a accesskey="u" href="../test-organization.html"><img src="../../../../../../../doc/src/images/up.png" alt="Up"></a><a accesskey="h" href="../../../index.html"><img src="../../../../../../../doc/src/images/home.png" alt="Home"></a><a accesskey="n" href="../testing-tools.html"><img src="../../../../../../../doc/src/images/next.png" alt="Next"></a>
|
|
</div>
|
|
</body>
|
|
</html>
|