boost/build
2
0
Fork 0
mirror of https://github.com/boostorg/build.git synced 2026-02-10 11:22:12 +00:00
Code Issues Projects Releases Wiki Activity

Updated the Boost Build test system documentation. Now all the command line options are described. Several function descriptions updated.

Browse Source 
[SVN r44983]
This commit is contained in:
Jurko Gospodnetić
2008-05-01 17:08:04 +00:00
parent f23d32f8b2
commit 21648313a7
1 changed files with 203 additions and 131 deletions
Download Patch File Download Diff File Expand all files Collapse all files

334
test/test_system.html
View File

@@ -12,14 +12,14 @@
<style type="text/css">
hr { color: black }
p.revision { text-align: right; font-style: italic }
pre.code { margin-left: 2em }
pre.example { margin-left: 2em; border: solid black thin }
pre.code { margin-left: 2em }
pre.example { margin-left: 2em; border: solid black thin }
pre.output { margin-left: 2em }
img.banner { border: 0; float: left }
h1 { text-align: right }
br.clear { clear: left }
div.attention { color: red }
</style>
</head>
@@ -34,6 +34,12 @@
<dl class="page-index">
<dt><a href="#sec-intro">Introduction for users</a></dt>
<dd>
<dl class="page-index">
<dt><a href="#sec-command-line-options">Command line options</a></dt>
</dl>
</dd>
<dt><a href="#sec-developers">Introduction for developers</a></dt>
<dd>
@@ -41,8 +47,8 @@
<dt><a href="#sec-intro-changing">Changing the working
directory</a></dt>
<dt><a href="#sec-intro-examining">Examining the working directory
and changing it</a></dt>
<dt><a href="#sec-intro-examining">Examining the working directory and
changing it</a></dt>
<dt><a href="#sec-intro-results">Test result</a></dt>
</dl>
@@ -87,14 +93,13 @@
<h2><a name="sec-intro">Introduction for users</a></h2>
<p>The testing system for Boost.Build is a small set of Python modules
and scripts for automatically testing user-obversable behaviour. It uses
components from testing systems of <a href=
"http://www.scons.org">Scons</a> and <a href=
"http://subversion.tigris.org">Subversion</a>, together with some
additional functionality.</p>
<p>The testing system for Boost.Build is a small set of Python modules and
scripts for automatically testing user-obversable behaviour. It uses
components from testing systems of <a href="http://www.scons.org">Scons</a>
and <a href="http://subversion.tigris.org">Subversion</a>, together with
some additional functionality.</p>
<p>To run the tests you'd need:</p>
<p>To run the tests you need to:</p>
<ol>
<li>Get the source tree of Boost.Build (located at <tt>tools/build</tt>
@@ -115,25 +120,47 @@
for testing.</li>
</ol>
<p>When all is done, you can run the tests with</p>
<p>When all is set, you can run all the tests using the <tt>test_all.py</tt>
script or you can run a specific test by starting its Python script
directly.</p>
<p>Examples:</p>
<pre class="code">
python test_all.py
</pre>
<p>which will use the default toolset, or you can specify toolset on the
command line, for example:</p>
<pre class="code">
python test_all.py borland
</pre>
<p>If everything's OK, you'll see a list of passed tests. Otherwise, a
failure will be reported.</p>
<p>It is possible to run a specific test, for example:</p>
<pre class="code">
python generators_test.py
</pre>
<p>If everything is OK, you will see a list of passed tests. Otherwise, a
failure will be reported.</p>
<h3><a name="sec-command-line-options">Command line options</a></h3>
<p>Test scripts will use the toolset you configured to be the default or
you can specify a specific one on the command line:</p>
<pre class="code">
python test_all.py borland
python generators_test.py msvc-7.1
</pre>
<p>Other test script flags you can specify on the command line are:</p>
<ul>
<li><tt>--default-bjam</tt> -- By default the test system will use the
Boost Jam executable found built in its default development build
location. This option makes it use the default one available on your
system, i.e. the one found in the system path.</li>
<li><tt>--preserve</tt> -- In case of a failed test its working
directory will be copied to the "failed_test" directory under the
current directory.</li>
<li><tt>--verbose</tt> -- Makes the test system and the run build system
display additional output. Note though that this may cause tests that
check the build system output to fail.</li>
</ul>
<h2><a name="sec-developers">Introduction for developers</a></h2>
<p>It is suggested that every new functionality come together with tests,
@@ -144,28 +171,28 @@ python generators_test.py
<li>For an interpreted language like Jam, without any static checks,
testing is simply the only sefeguard we can have.</li>
<li>Good tests allow to change internal design much more safely, and we
didn't nailed everything down yet.</li>
<li>Good tests allow us to change internal design much more safely, and we
have not gotten everything nailed down yet.</li>
</ul>
<p>Adding a new test is simple:</p>
<ol>
<li>Go to <tt>$boost_build_root/test/test_all.py</tt> and add new test
name to the list at the end of file. Suppose the test name is
"hello".</li>
name to the list at the end of the file. Suppose the test name is "hello".
</li>
<li>Add a new python module, in this example "hello.py", to do actual
<li>Add a new python module, in this example "hello.py", to do the actual
testing.</li>
</ol>
<p>The module, in general will perform these basic actions:</p>
<ol>
<li>Setting the initial working directory state</li>
<li>Set up the initial working directory state</li>
<li>
Running the build system and checking:
Run the build system and check the results:
<ol>
<li>generated output,</li>
@@ -176,10 +203,10 @@ python generators_test.py
</ol>
</li>
<li>Adding, removing or touching files, or changing their content and
then repeating the previous step, until satisfied.</li>
<li>Add, remove or touch files or change their content and then repeat
the previous step until satisfied.</li>
<li>Cleaning up</li>
<li>Clean up</li>
</ol>
<p>The "hello.py" module might contain:</p>
@@ -204,12 +231,12 @@ int main()
t.run_build_system()
# First, create a list of three pathnames
# First, create a list of three pathnames.
file_list = List("bin/$toolset/debug/") * List("hello.exe hello.obj")
# Second, assert that those files were added as result of the last build system invocation.
t.expect_addition(file_list)
# Invoke the build system once again
# Invoke the build system once again.
t.run_build_system("clean")
# Check if the files added previously were removed.
t.expect_removal(file_list)
@@ -221,18 +248,18 @@ t.cleanup()
<p>The <tt>test</tt> directory contains a file "template.py" which can be
used as a start for your own tests.</p>
<p>Overview of the most important methods of class
<tt>TestBoostBuild</tt> follows.</p>
<p>Overview of the most important methods of class <tt>Tester</tt> follows.
</p>
<h3><a name="sec-intro-changing">Changing the working directory</a></h3>
<p>The class <tt>TestBoostBuild</tt> creates a temporary directory in its
<p>The class <tt>Tester</tt> creates a temporary directory in its
constructor and changes to that directory. It can be modified by calling
these methods:</p>
<ul>
<li><tt>set_tree</tt> -- sets the content of the working directory to
be equal to the content of the specified directory. This method is
<li><tt>set_tree</tt> -- sets the content of the working directory to be
equal to the content of the specified directory. This method is
preferrable when directory tree for testing is large.</li>
<li><tt>write</tt> -- sets the content of file in a working directory.
@@ -245,28 +272,27 @@ t.cleanup()
<h3><a name="sec-intro-examining">Examining the working directory and
changing it</a></h3>
<p>The method <tt>read</tt>, inherited from the <tt>TestCmd</tt> class,
can be used to read any file in the working directory and check its
content. <tt>TestBoostBuild</tt> adds another method for tracking
changes. Whenever build system is run (via <tt>run_build_system</tt>),
the state of working dir before and after running is recorded. In
addition, difference between the two states -- i.e. lists of files that
were added, removed, modified or touched -- is stored in two member
variables, <tt>tree_difference</tt> and
<tt>unexpected_difference</tt>.</p>
<p>The method <tt>read</tt>, inherited from the <tt>TestCmd</tt> class, can
be used to read any file in the working directory and check its content.
<tt>Tester</tt> adds another method for tracking changes. Whenever the build
system is run (using <a href="#method-run_build_system"><tt>run_build_system
</tt></a>), the working dir state before and after running is recorded. In
addition, difference between the two states -- i.e. lists of files that were
added, removed, modified or touched -- are stored in two member variables -
<tt>tree_difference</tt> and <tt>unexpected_difference</tt>.</p>
<p>After than, the test author may specify that some change is expected,
for example, by calling <tt>expect_addition("foo")</tt>. This call will
check if the file was indeed added, and if so, will remove its name from
the list of added files in <tt>unexpected_difference</tt>. Likewise, it's
possible to specify that some changes are not interesting, for example a
call <tt>ignore("*.obj")</tt> will just remove every files with ".obj"
<p>After than, the test author may specify that some change is expected, for
example, by calling <tt>expect_addition("foo")</tt>. This call will check if
the file was indeed added, and if so, will remove its name from the list of
added files in <tt>unexpected_difference</tt>. Likewise, it is possible to
specify that some changes are not interesting, for example a call to
<tt>ignore("*.obj")</tt> will just remove every file with the ".obj"
extension from <tt>unexpected_difference</tt>.</p>
<p>When test has finished with expectations and ignoring, the member
<tt>unexpected_difference</tt> will contain the list of all changes not
yet accounted for. It is possible to assure that this list is empty by
calling <tt>expect_nothing_more</tt> member function.</p>
<tt>unexpected_difference</tt> will contain the list of all changes not yet
accounted for. It is possible to assure that this list is empty by calling
the <tt>expect_nothing_more</tt> member function.</p>
<h3><a name="sec-intro-results">Test result</a></h3>
@@ -277,24 +303,74 @@ t.cleanup()
<tt>pass_test</tt> and <tt>fail_test</tt> are used to explicitly give the
test outcome.</p>
<p>Typically, after test termination, the working directory is erased. To
debug a failed test, you should add "--preserve" option when invoking
test. On failure, the working directory will be copied to "failed_test"
directory in the current dir.</p>
<p>Typically, after test termination, the working directory is erased. See
the <a href="#sec-command-line-options">"--preserve" command line option</a>
for information on how to preserve the working directory content for failed
tests for debugging purposes.</p>
<h2 id="sec-reference">Reference documentation</h2>
<p>The test system is composed of class <tt>Tester</tt>, derived form
<tt>TestCmd.TestCmd</tt>, and helper class <tt>List</tt>. The methods of
<tt>Tester</tt>, and the class <tt>List</tt> are described below.</p>
<tt>TestCmd.TestCmd</tt>, and helper class <tt>List</tt>. <tt>Tester</tt>
and <tt>List</tt> methods are described below.</p>
<p>The documentation frequently refer to filename. In all cases, files
are specified in unix style: a sequence of components, separated by "/".
This is true on all platforms. In some contexts, a list of files is
allowed. In that case any object with sequence interface is allowed.</p>
<p>The documentation frequently refers to <tt>filename</tt>. In all cases,
files are specified in unix style: a sequence of components, separated by
"/". This is true on all platforms. In some contexts a list of files is
allowed. In those cases any object with a sequence interface is allowed.</p>
<h3><a name="method-__init__">Method <tt>__init__(self, workdir='',
arguments='', executable='bjam')</tt></a></h3>
<h3><a name="method-__init__">Method <tt>__init__(self, arguments="",
executable="bjam", match=TestCmd.match_exact, boost_build_path=None,
translate_suffixes=True, pass_toolset=True, use_test_config=True,
ignore_toolset_requirements=True, workdir="", **keywords)</tt></a></h3>
<p><b>Optional arguments:</b></p>
<ul>
<li><tt>arguments</tt>
- Arguments passed to the run executable.</li>
<li><tt>executable</tt>
- Name of the executable to invoke.</li>
<li><tt>match</tt>
- Function to use for compating actual and expected file contents.
</li>
<li><tt>boost_build_path</tt>
- Boost build path to be passed to the run executable.</li>
<li><tt>translate_suffixes</tt>
- Whether to update suffixes on the the file names passed from the
test script so they match those actually created by the current
toolset. For example, static library files are specified by using
the .lib suffix but when the 'gcc' toolset is used it actually
creates them using the .a suffix.</li>
<li><tt>pass_toolset</tt>
- Whether the test system should pass the specified toolset to the
run executable.</li>
<li><tt>use_test_config</tt>
- Whether the test system should tell the run executable to read in
the test_config.jam configuration file.</li>
<li><tt>ignore_toolset_requirements</tt>
- Whether the test system should tell the run executable to ignore
toolset requirements.</li>
<li><tt>workdir</tt>
- indicates an absolute directory where the test will be run from.
</li>
</ul>
<p><b>Optional arguments inherited from the base class:</b></p>
<ul>
<li><tt>description</tt>
- Test description string displayed in case of a failed test.</li>
<li><tt>subdir</tt>
- List of subdirectories to automatically create under the working
directory. Each subdirectory needs to be specified separately
parent coming before its child.</li>
<li><tt>verbose</tt>
- Flag that may be used to enable more verbose test system output.
Note that it does not also enable more verbose build system output
like the <a href="#sec-command-line-options">"--verbose" command
line option</a> does.</li>
</ul>
<p><b>Effects:</b></p>
@@ -302,20 +378,23 @@ t.cleanup()
<li>Remembers the current working directory in member
<tt>original_workdir</tt>.</li>
<li>Determines the location of executable (<code>bjam</code> by
<li>Determines the location of the executable (<code>bjam</code> by
default) and build system files, assuming that the current directory is
<tt>tools/build/test</tt>. Formulates jam invocation command, which
will include explicit setting for <tt>BOOST_BUILD_PATH</tt> variable
will include explicit setting for the <tt>BOOST_BUILD_PATH</tt> variable
and arguments passed to this methods, if any. This command will be used
by subsequent invocation of <a href=
"#method-run_build_system"><tt>run_build_system</tt></a>. Finally,
initializes the base class.</li>
by subsequent invocation of <a href="#method-run_build_system"><tt>
run_build_system</tt></a>. Finally, initializes the base class.</li>
<li>Changes current working dir to the temporary working directory
created by the base constructor.</li>
<li>Changes the current working directory to the temporary working
directory created by the base constructor.</li>
<li>If you want to run a test in a existing directory, pass it to
<li>If you want to run a test in an existing directory, pass it as
<tt>workdir</tt>.</li>
<li> Most parameters passed to this constructor function may be overruled
for each specific test system run (using <a href=
"#method-run_build_system"><tt> run_build_system</tt></a> parameters).
</ol>
<h3><a name="method-set_tree">Method <tt>set_tree(self,
@@ -353,7 +432,8 @@ t.cleanup()
relative paths.</p>
<h3><a name="method-run_build_system">Method <tt>run_build_system(self,
subdir='', extra_args='', stdout=None, stderr='', status=0,
extra_args="", subdir="", stdout=None, stderr="", status=0, match=None,
pass_toolset=None, use_test_config=None, ignore_toolset_requirements=None,
**kw)</tt></a></h3>
<p><b>Effects:</b></p>
@@ -396,104 +476,95 @@ t.cleanup()
<p>Read the specified file and returns it content. Raises an exception is
the file is absent.</p>
<h3><a name="method-read_and_strip">Method <tt>read_and_strip(self,
name)</tt></a></h3>
<h3><a name="method-read_and_strip">Method <tt>read_and_strip(self, name)
</tt></a></h3>
<p><b>Effects:</b></p>
<p>Read the specified file and returns it content, after removing
trailing whitespace from every line. Raises an exception is the file is
absent.</p>
<p>Read the specified file and returns it content, after removing trailing
whitespace from every line. Raises an exception is the file is absent.</p>
<p><b>Rationale:</b></p>
<p>Althought this method is questionable, there are a lot of cases when
jam or shells it uses insert spaces. It seems that introducing this
method is much simpler than dealing with all those cases.</p>
<p>Althought this method is questionable, there are a lot of cases when jam
or shells it uses insert spaces. It seems that introducing this method is
much simpler than dealing with all those cases.</p>
<h3><a name="methods-expectations">Methods for declaring
expectations</a></h3>
<h3><a name="methods-expectations">Methods for declaring expectations</a>
</h3>
<p>Accordingly to the number of changes kinds that are detected, there
are four methods that specify that test author expects a specific change
to occur. They check <tt>self.unexpected_difference</tt>, and if the
change is present there, it is removed. Otherwise, test fails.</p>
<p>Accordingly to the number of changes kinds that are detected, there are
four methods that specify that test author expects a specific change to
occur. They check <tt>self.unexpected_difference</tt>, and if the change is
present there, it is removed. Otherwise, test fails.</p>
<p>Each method accepts a list of names. Those names use <tt>/</tt> path
separator on all systems. Additionaly, the test system translates
suffixes appropriately. For the test to be portable, suffixes should use
Windows convention: <tt>exe</tt> for executables, <tt>dll</tt> for
dynamic libraries and <tt>lib</tt> for static libraries. Lastly, the
string "$toolset" in file names is replaced by the name of tested
toolset.</p>
separator on all systems. Additionaly, the test system translates suffixes
appropriately. For the test to be portable, suffixes should use Windows
convention: <tt>exe</tt> for executables, <tt>dll</tt> for dynamic libraries
and <tt>lib</tt> for static libraries. Lastly, the string "$toolset" in file
names is replaced by the name of tested toolset.</p>
<p><b>Note:</b> The <tt>List</tt> helper class might be useful to create
lists of names.</p>
<p><b>Note:</b> The file content can be examined using
<p><b>Note:</b> The file content can be examined using the
<tt>TestCmd.read</tt> function.</p>
<p>The members are:</p>
<ul>
<li>expect_addition</li>
<li>expect_removal</li>
<li>expect_modification</li>
<li>expect_nothing</li>
</ul>
<p>Note that <tt>expect_modification</tt> is used to check that a either
file content or timestamp has changed. The rationale is that some
compilers change content even if sources does not change, and it's easier
to have a method which checks for both content and time changes.</p>
file content or timestamp has changed. The rationale is that some compilers
change content even if sources does not change, and it's easier to have a
method which checks for both content and time changes.</p>
<p>There's also a member <tt>expect_nothing_more</tt>, which checks that
all the changes are either expected or ignored, in other words that
<p>There's also a member <tt>expect_nothing_more</tt>, which checks that all
the changes are either expected or ignored, in other words that
<tt>unexpected_difference</tt> is empty by now.</p>
<p>Lastly, there's a method to compare file content with expected
content:</p>
<tt>expect_content(self, name, content, exact=0)</tt>
<tt>expect_content(self, name, content, exact=0)</tt>
<p>The method fails the test if the content of file identified by 'name'
is different from 'content'. If 'exact' is true, the file content is used
<p>The method fails the test if the content of file identified by 'name' is
different from 'content'. If 'exact' is true, the file content is used
as-is, otherwise, two transformations are applied:</p>
<ul>
<li>The <tt>read_and_strip</tt> method is used to read the file, which
removes trailing whitespace</li>
<li>Each backslash in the file content is converted to forward
slash.</li>
<li>Each backslash in the file content is converted to forward slash.</li>
</ul>
<h3><a name="methods-ignoring">Methods for ignoring changes</a></h3>
<p>There are five methods which ignore changes made to the working tree.
They silently remove elements from <tt>self.unexpected_difference</tt>,
and don't generate error if element is not found. They accept shell style
They silently remove elements from <tt>self.unexpected_difference</tt>, and
don't generate error if element is not found. They accept shell style
wildcard.</p>
<p>The following methods correspond to four kinds of changes:</p>
<ul>
<li>ignore_addition(self, wildcard)</li>
<li>ignore_removal(self, wildcard)</li>
<li>ignore_modification(self, wildcard)</li>
<li>ignore_touch(self, wilcard)</li>
</ul>
<p>The method <tt>ignore(self, wildcard)</tt> ignores all the changes
made to files that match a wildcard.</p>
<p>The method <tt>ignore(self, wildcard)</tt> ignores all the changes made
to files that match a wildcard.</p>
<h3><a name="methods-result">Methods for explicitly specifying
results</a></h3>
<h3><a name="methods-result">Methods for explicitly specifying results</a>
</h3>
<h4>Method <tt>pass_test(self, condition=1)</tt></h4>
@@ -507,7 +578,7 @@ t.cleanup()
true.</p>
<h3><a name="class-list">Helper class <tt>List</tt></a></h3>
The class has sequence interface and two additional methods.
The class has sequence interface and two additional methods.
<h4>Method <tt>__init__(self, string)</tt></h4>
@@ -529,8 +600,8 @@ t.cleanup()
<pre>
l = "a b" * List("c d")
for e in l:
print e
print e
</pre>
<p>will output</p>
@@ -539,13 +610,14 @@ t.cleanup()
ad
bc
bd
</pre>
<hr>
<p class="revision">Last modified: Mar 11, 2005</p>
<p class="revision">Last modified: May 01, 2008</p>
<p>&copy; Copyright Vladimir Prus 2002, 2003, 2004, 2005.
<p>&copy; Copyright Vladimir Prus 2002, 2003, 2004, 2005.<br>
&copy; Copyright Jurko Gospodnetic 2008.<br>
Distributed under the Boost Software License, Version 1.0.
(See accompanying file LICENSE_1_0.txt or http://www.boost.org/LICENSE_1_0.txt)</p>
</body>