mirror of
https://github.com/boostorg/test.git
synced 2026-02-18 14:32:11 +00:00
170 lines
10 KiB
XML
170 lines
10 KiB
XML
<?xml version="1.0" encoding="utf-8"?>
|
|
<!DOCTYPE section PUBLIC "-//Boost//DTD BoostBook XML V1.0//EN" "../../../../tools/boostbook/dtd/boostbook.dtd" [
|
|
<!ENTITY utf "<acronym>UTF</acronym>">
|
|
]>
|
|
<section id="tutorial.intro-in-testing">
|
|
<sectioninfo>
|
|
<author>
|
|
<firstname>John</firstname>
|
|
<othername role="mi">R</othername>
|
|
<surname>Phillips</surname>
|
|
<email>jphillip at capital dot edu (please unobscure)</email>
|
|
</author>
|
|
<copyright>
|
|
<year>2006</year>
|
|
<holder>John R. Phillips</holder>
|
|
</copyright>
|
|
|
|
<legalnotice>
|
|
<simpara>
|
|
Use, modification and distribution is subject to the Boost Software License, Version 1.0. (See accompanying file
|
|
<filename>LICENSE_1_0.txt</filename> or copy at
|
|
<ulink url="http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt</ulink> )
|
|
</simpara>
|
|
</legalnotice>
|
|
</sectioninfo>
|
|
|
|
<title>Introduction into testing … or why testing is worth the effort</title>
|
|
<titleabbrev>Introduction into testing</titleabbrev>
|
|
|
|
<para role="first-line-indented">
|
|
For almost everyone, the first introduction to the craft of programming is a version of the simple "Hello World" program. In C++, this first example might be written as
|
|
</para>
|
|
|
|
<btl-snippet name="snippet7"/>
|
|
|
|
<para role="first-line-indented">
|
|
This is a good introduction for several reasons. One is that the program is short enough, and the logic of its
|
|
execution simple enough that direct inspection can show whether it is correct in all use cases known to the new
|
|
student programmer. If this were the complexity of all programming, there would be no need to test anything before
|
|
using it. In programming as a new student experiences it, testing is pointless and adds unneeded complexity.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
However, no actual programs are as simple as an introductory lesson makes "Hello World" seem. Not even "Hello World".
|
|
In all real programs, there are decisions to be made and multiple paths of execution based on these decisions. These
|
|
decisions could be based on user input, streaming data, resource availability and dozens of other factors. The
|
|
programmer strives to control the inputs, and results of these decisions, but no one can keep all of them clearly
|
|
in mind once the size of the project exceeds just a few hundred lines. Even "Hello World" hides complexities of
|
|
this sort in the simple seeming call to std::cout.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
Since the individual programmer can no longer determine the correctness of the program, there is a need for a
|
|
different approach. An obvious possibility is testing the program after construction. Someone develops a set of
|
|
test cases, where inputs are given to the program such that the behavior and outputs of a correctly performing
|
|
program are known. The performance of the new program is compared to known standards and the new program either
|
|
passes or fails. If it fails, attempts are made to fix it. If the test cases are carefully chosen, the specifics of
|
|
the failure give an indication of what in the program needs to be fixed.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
This is an improvement over just not knowing whether the program is working properly, but it isn't a big improvement.
|
|
If the whole program is tested at once, it is nearly impossible to develop test cases that clearly indicate what
|
|
the failure is. The system is too complex, and the programmer still needs to understand almost all of the possible
|
|
outcomes to be able to develop tests. As always, when a problem is too big and complicated a good idea is to try
|
|
splitting it into smaller and simpler pieces.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
This approach leads to a layered system of testing, that is similar to the layered approach to original development
|
|
and should be integrated into it. When writing a program, the design is factored into small units that are
|
|
conceptually and structurally easier to grasp. A standard rule for this is that one unit performs one job or
|
|
embodies one concept. These simple units are composed into larger and more complicated algorithms by passing needed
|
|
information into a unit and receiving the desired result out of it. The units are integrated to perform the whole
|
|
task. Testing should reflect this structure of development.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
The simplest layer is Unit Testing. A unit is the smallest conceptually whole segment of the program. Examples of
|
|
basic units might be a single class or a single function. For each unit, the tester (who may or may not be the
|
|
programmer) attempts to determine what states the unit can encounter while executing as part of the program. These
|
|
states include determining the range of appropriate inputs to the unit, determining the range of possible
|
|
inappropriate inputs, and recognizing any ways the state of the rest of the program might affect execution in this
|
|
unit.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
With so many general statements, an example will help clarify. Imagine the following procedural function is part of
|
|
a program, and the programmer wants to test it. For the sake of brevity, header includes and namespace qualifiers
|
|
have been suppressed.
|
|
</para>
|
|
|
|
<btl-snippet name="snippet8"/>
|
|
|
|
<para role="first-line-indented">
|
|
This code, although brief and simple is getting long enough that it takes attention to find what is done and why.
|
|
It is no longer obvious at a glance what the intent of the program is, so careful naming must be used to carry that
|
|
intent.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
Thanks to the control structures, there are some obvious execution paths in the code. However, there are also a few
|
|
less obvious paths. For example, if the root finder takes many steps to converge to an acceptable answer, the
|
|
vector that is holding the history of steps taken may need to reallocate for additional space. In this case, there
|
|
are many hidden steps in the single push_back command. These steps also include the chance of failure, since that
|
|
is always a possibility in a memory allocation.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
A second example notes that the value of the function at the low guess has not been tested, so there is the chance
|
|
of a zero division. Also, if the value of the function at the high guess is zero, the root finder will miss that
|
|
root entirely. It may even fall into an infinite loop if no root lies between the low and high values.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
In this unit, proper testing includes checking the behavior in each possibility. It also includes checking the
|
|
function by giving inputs where the correct answer is known and checking the results against that answer. Thus,
|
|
the unit is tested in every execution path to assure proper behavior.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
Test cases are chosen to expose as many errors as possible. A defining characteristic of a good test case is that
|
|
the programmer knows what the unit should do if it is functioning properly. Test cases should be generated to
|
|
exercise each available execution path. For the above snippet, this includes the obvious and the not so obvious
|
|
paths. Every path should be tested, since every path is a possible outcome of program execution.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
Thus, to write a good testing suite, the tester must know the structure of the code. The most dependable way to
|
|
accomplish this is if the original programmer writes tests as part of creating the code. In fact, it is advisable
|
|
that the tests are produced before the code is written, and updated whenever structure decisions are changed. This
|
|
way, the tests are written with a view toward how the unit should perform instead of reproducing the programmer's
|
|
thinking from writing the code. While black box testing is also useful, it is important that someone who knows the
|
|
design decisions made and the rationale for those decisions test the code unit. A programmer who can't devise
|
|
good tests for a unit does not yet know the problem at hand well enough to program dependably.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
When a unit is completed and tested, it is ready for integration with other units in the program. This is
|
|
integration should also be tested. At this point, the test cases focus on the interaction between the units. Tests
|
|
are designed to exercise each way the units can affect each other.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
This is the point in development where proper unit testing really shines. If each unit is doing what it should be
|
|
doing and not creating unexpected side effects, any issues in testing a set of integrated units must come from how
|
|
they are passing information. Thus, the nearly intractable problem of finding an error while many units interact
|
|
becomes the less intimidating problem of finding the breakdown in communications.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
At each layer of increasing complexity, new tests are run, and if the prior tests of the components are well
|
|
designed and all issues are fixed, new errors are isolated to the integration. This process continues, in parallel
|
|
with development, from the smallest units to the completed program.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
This shows that there is a need to be able to check and test code snippets such as individual functions and classes
|
|
independent the program of which they will become a part. That is, the need for a means to provide predetermined
|
|
inputs to the unit to check the outputs against expected results. Such a system must allow for both normal
|
|
operation and error conditions, allow the programmer to produce a thorough description of the results.
|
|
</para>
|
|
|
|
<para role="first-line-indented">
|
|
This is the goal and rationale for all unit testing, and supporting testing of this sort is the purpose of the
|
|
Boost.Test library. As is shown below, Boost.Test provides a well-integrated set of tools to support this testing
|
|
effort throughout the programming and maintenance cycles of software development.
|
|
</para>
|
|
</section>
|