build/v2/doc/src/extending.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE appendix PUBLIC "-//Boost//DTD BoostBook XML V1.0//EN"
  "http://www.boost.org/tools/boostbook/dtd/boostbook.dtd">

  <chapter id="bbv2.extender">
    <title>Extender Manual</title>

    <section id="bbv2.extender.intro">
      <title>Introduction</title>

  <para>This document explains how to extend Boost.Build to accomodate
  your local requirements. Let's start with quite simple, but
  realistic example.</para>

  <para>Say you're writing an application which generates C++ code. If
  you ever did this, you know that it's not nice. Embedding large
  portions of C++ code in string literals is very awkward. A much
  better solution is:</para>

  <orderedlist>
    <listitem>
      <simpara>
        Write the template of the code to be generated, leaving
    placeholders at the points which will change
      </simpara>
    </listitem>

    <listitem>
      <simpara>
        Access the template in your application and replace
    placeholders with appropriate text.
      </simpara>
    </listitem>

    <listitem>
      <simpara>Write the result.</simpara>
    </listitem>
  </orderedlist>

  <para>It's quite easy to archive. You write special verbatim files,
  which are just C++, except that the very first line of the file
  gives a name of variable that should be generated. A simple tool
  is created which takes verbatim file and creates a cpp file with
  a single char* variable, which name is taken from the first line
  of verbatim file, and which value is properly quoted content of
  the verbatim file.</para>

  <para>Let's see what Boost.Build can do.</para>

  <para>First off, Boost.Build has no idea about "verbatim files". So,
  you must register a new type. The following code does it:</para>

<programlisting>
import type ;
type.register VERBATIM : verbatim ;
</programlisting>

  <para>The first parameter to 'type.register' gives the name of
  declared type. By convention, it's uppercase. The second
  parameter is suffix for this type. So, if Boost.Build sees
  "code.verbatim" in the list of sources, it knows that it's of
  type <literal>VERBATIM</literal>.</para>

  <para>Lastly, you need a tool to convert verbatim files to C++. Say
  you've sketched such a tool in Python. Then, you have to inform
  Boost.Build about the tool. The Boost.Build concept which
  represents a tool is <emphasis>generator</emphasis>.</para>

  <para>First, you say that generator 'inline-file' is able to convert
  VERBATIM type into C++:</para>

<programlisting>
import generators ;
generators.register-standard verbatim.inline-file : VERBATIM : CPP ;
</programlisting>

  <para>Second, you must specify the commands to be run to actually
  perform convertion:</para>

<programlisting>
actions inline-file
{
    "./inline-file.py" $(&lt;) $(&gt;)
}
</programlisting>
<!-- We use verbatim.inline-file in one place and just inline-file in
             another. Is this confusing for user?
        -->

  <para>Now, we're ready to tie it all together. Put all the code
  above in file "verbatim.jam", add "import verbatim ;" to
  "project-root.jam", and it's possible to write the following in
  Jamfile:</para>

<programlisting>
exe codegen : codegen.cpp class_template.verbatim usage.verbatim ;
</programlisting>

  <para>
The verbatim files will be automatically converted into C++
and linked it.
  </para>

      <para>In the subsequent sections, we will extend this example, and review
        all the mechanisms in detail. The complete code is available in <ulink url=
          "../../example/customization">example/customization</ulink>
        directory.
      </para>

  </section>
  <section id="bbv2.extending.targets">
    <title>Target types</title>
      <para>The first thing we did in the <link
          linkend="bbv2.extender.intro">intruduction</link> was declaring a
      new target type:
<programlisting>
import type ;
type.register VERBATIM : verbatim ;
</programlisting>
        The type is the most important property of a target. Boost.Build can
        automatically generate necessary build actions only because you
        specify the desired type (using the different main target rules), and
        because Boost.Build can guess the type of sources from their
        extensions.        
      </para>

      <para>The first two parameters for the <code>type.register</code> rule
        are the name of new type and the list of extensions associated with
        it. A file with an extension from the list will have the given target
        type. In the case where a target of the declared type is generated
        from other sources, the first specified extension will be used. This
        behaviour can be changed using the
        <code>type.set-generated-target-suffix</code> rule.
      </para>

      <para>
        Something about 'main' types.
      </para>

      <para>Something about base types.
      </para>

      <section id="bbv2.extending.scanners">
        <title>Scanners</title>
        <para>
          Sometimes, a file can refer to other files via some include
          mechanism. To make Boost.Build track dependencies to the included
          files, you need to provide a scanner. The primary limitation is that
          only one scanner can be assigned to a target type.
        </para>

        <para>First, we need to declare a new class for the scanner:
<programlisting>
class verbatim-scanner : common-scanner
{
    rule pattern ( )
    {
        return "//###include[ ]*\"([^\"]*)\"" ;
    }
}
</programlisting>         
          All the complex logic is in the <code>common-scanner</code> class,
          and you only need to override the method which returns the regular
          expression to be used for scanning. The paranthethis in the regular
          expression indicate which part of the string is the name of the
          included file. 
        </para>

        <para>After that, we need to register our scanner class:
<programlisting>
scanner.register verbatim-scanner : include ;
</programlisting>
            The value of the second parameter, in this case
            <code>include</code>, specifies which properties contain the list
            of paths which should be searched for the included files.
         </para>

        <para>Finally, we assign the new scaner to the <code>VERBATIM</code>
        target type:
<programlisting>
type.set-scanner VERBATIM : verbatim-scanner ;
</programlisting>
          That's enough for scanning include dependencies.
        </para>

      </section>


  </section>
  
  <section id="bbv2.extending.tools">
    <title>Tools and generators</title>
      <para>
        This section will describe how Boost.Build can be extended to support
        new tools.
      </para>

      <para>For each additional tool, a Boost.Build object called generator
        must be created. That object has specific types of targets which it
        accepts an produces. Using that information, Boost.Build is able
        to automatically invoke the generator. For example, if you declare a
        generator which takes a target of the type <literal>D</literal> and
        produces a target of the type <literal>OBJ</literal>, when placing a
        file with extention <literal>.d</literal> in a list of sources will
        cause Boost.Build to invoke your generator, and then to link the
        resulting object file into an application. (Of course, this requires
        that you specify that the <literal>.d</literal> extension corresponds
        to the <literal>D</literal> type.)
      </para>

      <para>Each generator should be an instance of a class derived from the
        <code>generator</code> class. In the simplest case, you don't need to
        create a derived class, but simply create an instance of the
        <code>generator</code> class. Let's review the example we've seen in the
        <link linkend="bbv2.extender.intro">introduction</link>.
<programlisting>
import generators ;
generators.register-standard verbatim.inline-file : VERBATIM : CPP ;
actions inline-file
{
    "./inline-file.py" $(&lt;) $(&gt;)
}
</programlisting>
      </para>

      <para>We declare a standard generator, specifying its id, the source type
        and the target type. When invoked, the generator will create a target
        of type <literal>CPP</literal> which will have the source target of
        type <literal>VERBATIM</literal> as the only source. But what command
        will be used to actually generate the file? In bjam, actions are
        specified using named "actions" blocks and the name of the action
        block should be specified when creating targets. By convention,
        generators use the same name of the action block as their own id. So,
        in above example, the "inline-file" actions block will be use to
        convert the source into the target.
      </para>

      <para>
        There are two primary kinds of generators: standard and composing,
        which are registered with the
        <code>generators.register-standard</code> and the
        <code>generators.register-composing</code> rules, respectively. For
        example:
<programlisting>
generators.register-standard verbatim.inline-file : VERBATIM : CPP ;
generators.register-composing mex.mex : CPP LIB : MEX ;
</programlisting>
        The first generators takes a <emphasis>single</emphasis> source of type
        <code>VERBATIM</code> and produces a result. The second generator
        takes any number of sources, which can have either the
        <code>CPP</code> or the <code>LIB</code> type. Composing generators
        are typically used for generating top-level target type. For example,
        the first generator invoked when building an <code>exe</code> target
        is a composing generator corresponding to the proper linker.
      </para>

      <para>You should also know about two specific function for registering
        generators: <code>generators.register-c-compiler</code> and
        <code>generators.register-linker</code>. The first sets up header
        dependecy scanning for C files, and the seconds handles various
        complexities like searched libraries. For that reason, you should always
        use those functions when adding support for compilers and linkers.
      </para>

      <para>(Need a note about UNIX)</para>

      <bridgehead>Custom generator classes</bridgehead>

      <para>The standard generators allows you to specify source and target
        types, action, and a set of flags. If you need anything more complex,
        you need to create a new generator class with your own logic. Then,
        you have to create an instance of that class and register it. Here's
        an example how you can create your own generator class:
<programlisting>
class custom-generator : generator
{
    rule __init__ ( * : * )
    {
        generator.__init__ $(1) : $(2) : $(3) : $(4) : $(5) : $(6) : $(7) : $(8) : $(9) ;
    }
}

generators.register 
  [ new custom-generator verbatim.inline-file : VERBATIM : CPP ] ;
</programlisting>
        This generator will work exactly like the
        <code>verbatim.inline-file</code> generator we've defined above, but
        it's possible to customize the behaviour by overriding methods of the
        <code>generator</code> class.
      </para>

      <para>There are two methods of interest. The <code>run</code> methods is
        responsible for overall process - it takes a number of source targets,
        converts them the the right types, and creates the result. The
        <code>generated-targets</code> method is called when all sources are
        converted to the right types to actually create the result.
      </para>

      <para>The <code>generated-target</code> method can be overridden when you
        want to add additional properties to the generated targets or use
        additional sources. For example (which is real), you have a tool for
        analysing programs, which should be given a name of executable and the
        list of all sources. Naturally, you don't want to list all source
        files manually. Here's how the <code>generated-target</code> method
        can find the list of sources automatically:
<programlisting>
class itrace-generator : generator {
....
    rule generated-targets ( sources + : property-set : project name ? )
    {
        local leafs ;
        local temp = [ virtual-target.traverse $(sources[1]) : : include-sources ] ;
        for local t in $(temp)
        {
            if ! [ $(t).action ]
            {
                leafs += $(t) ;
            }
        }
        return [ generator.generated-targets $(sources) $(leafs)
          : $(property-set) : $(project) $(name) ] ;
    }
}
generators.register [ new itrace-generator nm.itrace : EXE : ITRACE ] ;
</programlisting>
        The <code>generated-targets</code> rule will be called with a single
        source target of type <literal>EXE</literal>. The call to the
        <code>virtual-target.traverse</code> will return all targets the
        executable depends on, and we further find files which are not
        produced from anything. The found targets are added to the sources.
      </para>
      
      <para>The <code>run</code> method can be overriden to completely
        customize the way generator works. In particular, the conversion of
        sources to the desired types can be completely customized. Here's
        another real example. Tests for the Boost Python library usually
        consist of two parts: a Python program and a C++ file. The C++ file is
        compiled to Python extension which is loaded by the Python
        program. But in the likely case that both files have the same name,
        the created Python extension must be renamed. Otherwise, Python
        program will import itself, not the extension. Here's how it can be
        done:
<programlisting>
rule run ( project name ? : property-set : sources * : multiple ? )
{       
    local python ;
    for local s in $(sources)
    {
        if [ $(s).type ] = PY
        {
            python = $(s) ;
        }
    }
    
    local libs ;
    for local s in $(sources)
    {
        if [ type.is-derived [ $(s).type ] LIB ]
        {
            libs += $(s) ;
        }
    }
    
    local new-sources ;
    for local s in $(sources)
    {
        if [ type.is-derived [ $(s).type ] CPP ] 
        {
            local name = [ $(s).name ] ;
            if $(name) = [ $(python).name ] 
            {
                name = $(name)_ext ;
            }                                
            new-sources += [ generators.construct $(project) $(name) :
              PYTHON_EXTENSION : $(property-set) : $(s) $(libs) ] ;
        }
    }
        
    result = [ construct-result $(python) $(new-sources) : $(project) $(name) 
                 : $(property-set) ] ;        
}    
</programlisting>        
        First, we separate all source into python files, libraries and C++
        sources. For each C++ source we create a separate Python extension by
        calling <code>generators.construct</code> and passing the C++ source
        and the libraries. At this point, we also change the extension's name,
        if necessary.        
      </para>

 
    </section>

    <section id="bbv2.extending.features">
      <title>Features</title>
      <para>
        Often, we need to control the options passed the invoked tools. This 
        is done with features. Consider an example:
<programlisting>
# Declare a new feature
import feature : feature ;
feature verbatim-options : : free ;

# Cause the value of the 'verbatim-options' feature to be
# available as 'OPTIONS' variable inside verbatim.inline-file
import toolset : flags ;
flags verbatim.inline-file OPTIONS &lt;verbatim-options&gt; ;

# Use the "OPTIONS" variable
actions inline-file
{
    "./inline-file.py" $(OPTIONS) $(&lt;) $(&gt;)
}
</programlisting>
        We first define a new feature. Then, the <code>flags</code> invocation
        says that whenever verbatin.inline-file action is run, the value of
        the <code>verbatim-options</code> feature will be added to the
        <code>OPTIONS</code> variable, an can be used inside the action body.
        You'd need to consult online help (--help) to find all the features of
        the <code>toolset.flags</code> rule.
      </para>

      <para>
        There are two ways to control a tool using features. The
        first is to declare a "generic" feature, which value is passed to the
        tool without modification. For example, the value of the
        <code>cxxflags</code> feature ends up in the compiler's command
        line. Such a generic feature allows the user to get full control and
        should be always provided when you write a tool.
      </para>

      <para>The second approach is to declare a more specific feature, 
        which is directly translated to specific options. For example, 
        the <code>&lt;optimization&gt;speed</code> property might add
        <literal>-O3</literal> to the compiler's command line. Specific
        features are preferred over generic for a couple of reasons:

        <itemizedlist>
          <listitem>
            <para>For a specific feature, you will be able to build
            your project with two different settings of the feature. If you
            pass compiler options, Boost.Build assumes you know what you are
            doing, and would not care about the options.
            </para>
          </listitem>

          <listitem>
            <para>If a feature makes sense for more than one tool, using
              feature will make your Jamfiles more portable, as opposed to 
              passing specific flags.              
            </para>
          </listitem>
        </itemizedlist>
      </para>

       
      <bridgehead>Steps for adding a feauture</bridgehead>
      <para>Adding a feature requires three steps:

        <orderedlist>
          <listitem><para>Declaring a feature. For that, the "feature.feature"
              rule is used. You should have to decide on the set of <link
              linkend="bbv2.reference.features.attributes">feature
              attributes</link>:

              <itemizedlist>
                <listitem><para>if feature has several values, and
                    significally affects build, make it "propagated", so that
                    whole project is build with the same value by
                    default</para></listitem> 

                <listitem><para>if a feature does not have a fixed list of
                    values, it must be "free".</para></listitem>

                <listitem><para>if feature is used to refer to a path, it must
                be "path".</para></listitem>
        
                <listitem><para>if feature is used to refer to some target, it
                must be "dependency".</para></listitem>
              </itemizedlist>
              </para>
          </listitem>
              

          <listitem><para>Converting the feature value into variable. To use
              feature in build action, it must be converted into a variable,
              accessible in build action. This is accomplished by
              "toolset.flags" rule.</para></listitem>

      
          <listitem><para>Using the variable. The variable set in step 2 can
              be used in build action to form command parameters or
              files.</para></listitem>

        </orderedlist>
      </para>

      <bridgehead>Another example</bridgehead>

      <para>Here's an another example.
        Let's see how we can make a feature which refers to a target. For example,
        when linking dynamic libraries on windows, one sometimes needs to specify
        "DEF file", telling what functions should be exported. It would be nice to
        use this file like this:
<programlisting>
        lib a : a.cpp : &lt;def-file&gt;a.def ;
</programlisting>
        Actually, this feature is already supported, but anyway...
      </para>

      <orderedlist>
        <listitem>
          <para>Since the feature refers to a target, it must be "dependency".
<programlisting>
feature def-file : : free dependency ;
</programlisting>
            </para></listitem>

        <listitem><para>One of the toolsets which cares about DEF files is
msvc. The following line should be added to it.

<programlisting>
flags msvc.link DEF_FILE &lt;def-file&gt; ;
</programlisting>
            </para></listitem>

        <listitem><para>Since the DEF_FILE variable is not used by the
msvc.link action, we need to modify it to be:

<programlisting>
actions link bind DEF_FILE
{
    $(.LD) .... /DEF:$(DEF_FILE) ....
}
</programlisting>
            </para>
          
          
          <para> Note the "bind DEF_FILE" part. It tells bjam that DEF_FILE
            refers to a file, otherwise the variable will contain internal
            target name, which is not likely to make sense for the linker.
          </para>

          <para>
            We've almost done, but should stop for a small workaround. Add the following
            code to msvc.jam

<programlisting>
rule link
{
    DEPENDS $(&lt;) : [ on $(&lt;) return $(DEF_FILE) ] ;
}
</programlisting>

            This is needed to accomodate some bug in bjam, which hopefully
            will be fixed one day.</para></listitem>

      </orderedlist>

      <bridgehead>Variants and composite features.</bridgehead>

      <para>Sometimes you want to create a shorcut for some set of
        features. For example, <code>release</code> is a value of the
        <code>variant</code> and is a shortcut for a set of features.
        </para>.

      <para>It is possible to define your build variants. For example:
<programlisting>
variant crazy : &lt;optimization&gt;speed &lt;inlining&gt;off
                &lt;debug-symbols&gt;on &lt;profiling&gt;on ;
</programlisting>
        will define a new variant with the specified set of properties. You
        can also extend an existing variant:
<programlisting>
variant super_release : release : &lt;define&gt;USE_ASM ;
</programlisting>
        In this case, <code>super_release</code> will expand to all properties
        specified by <code>release</code>, and the additional one you've specified.
      </para>

      <para>You are not restricted to using the <code>variant</code> feature
      only. Here's example which defines a brand new feature:
<programlisting>
feature parallelism : mpi fake none : composite link-incompatible ;
feature.compose &lt;parallelism&gt;mpi : &lt;library&gt;/mpi//mpi/&lt;parallelism&gt;none ;
feature.compose &lt;parallelism&gt;fake : &lt;library&gt;/mpi//fake/&lt;parallelism&gt;none ;
</programlisting>
        This will allow you to specify value of feature
        <code>parallelism</code>, which will expand to link to the necessary
        library. 
      </para>
     
  </section>
  
  <section id="bbv2.extending.rules">
    <title>Main target rules</title>
    <para>
      The main target rule is what creates a top-level target, for example "exe" or
      "lib". It's quite likely that you'll want to declare your own and
      there are as many as three ways to do that.
    </para>
    
    <para>The first is the simplest, but is sufficient in a number of
      cases. Just write a wrapper rule, which will redirect to any of the
      existing rules. For example, you have only one library per directory and
      want all cpp files in the directory to be compiled. You can achieve this
      effect with:
<programlisting>
lib codegen : [ glob *.cpp ] ;
</programlisting>
      but what if you want to make it even simple. Then, you add the following
      definition to the project-root.jam file:
<programlisting>
rule glib ( name : extra-sources * : requirements * )
{
    lib $(name) : [ glob *.cpp ] $(extra-sources) : $(requirements) ;
}
</programlisting>
which would allow to reduce Jamfile to
<programlisting>
glib codegen ;
</programlisting>
    </para>

    <para>The second approach is suitable when your target rule should just
      produce a target of specific type. Then, when declaring a type you
      should tell Boost.Build that a main target rule should be created.
      For example, if you create a module "obfuscate.jam" containing:

<programlisting>
import type ;
type.register OBFUSCATED_CPP  : ocpp : : main ;

import generators ;
generators.register-standard obfuscate.file : CPP : OBFUSCATED_CPP ;
</programlisting>
      and import that module, you'll be able to use the rule "obfuscated-cpp"
      in Jamfiles, which will convert source to the OBFUSCATED_CPP type.
    </para>

    <para>
      The remaining method is to declare your own main target class. The
      simplest example of this can be found in "build/alias.jam" file. The
      current V2 uses this method when transformations are relatively
      complex. However, we might deprecate this approach. If you find that you
      need to use it (that is, the first two approaches are not sufficient),
      please let us know by posting to the mailing list.
    </para>
    

  </section>

  <section id="bbv2.extending.toolset_modules">

    <title>Toolset modules</title>

    <para>If your extensions will be used only on one project, they can be
      placed in a separate <filename>.jam</filename> file which will be
      imported by your <filename>project-root.jam</filename>. If the
      extensions will be used on many projects, the users will thank you for 
      a finishing touch.
    </para>

    <para>The standard way to use a tool in Boost.Build is the
      <code>using</code> rule. To make it work, you module should provide an
      <code>init</code> rule. The rule will be called with the same parameters
      which were passed to the <code>using</code> rule. The set of allowed
      parameters is determined by you. For example, you can allow the user to
      specify paths, tool version, or tool options.
    </para>

    <para>Here are some guidelines which help to make Boost.Build more
      consistent:
      <itemizedlist>
        <listitem><para>The <code>init</code> rule should never fail. Even if
          user provided a wrong path, you should emit a warning and go
          on. Configuration may be shared between different machines, and
          wrong values on one machine can be OK on another.
          </para></listitem>

        <listitem><para>Prefer specifying command to be executed to specifying
            path. First of all, this gives more control: it's possible to
            specify
<programlisting>
/usr/bin/g++-snapshot
time g++
</programlisting>
            as the command. Second, while some tools have a logical
            "installation root", it better if user don't have to remember if
            a specific tool requires a full command or a path.            
          </para></listitem>

        <listitem><para>Check for multiple initialization. A user can try to
            initialize the module several times. You need to check for this
            and decide what to do. Typically, unless you support several
            versions of a tool, duplicate initialization is a user error. If
            tool version can be specified during initialization, make sure the
            version is either always specified, or never specified (in which
            case the tool is initialied only once). For example, if you allow:
<programlisting>
using yfc ;
using yfc : 3.3 ;
using yfc : 3.4 ;
</programlisting>
            Then it's not clear if the first initialization corresponds to
            version 3.3 of the tool, version 3.4 of the tool, or some other
            version. This can lead to building twice with the same version.
            </para></listitem>

        <listitem><para>If possible, the <code>init</code> must be callable
          with no parameters. In which case, it should try to autodetect all
          the necessary information, for example, by looking for a tool in
          <envar>PATH</envar> or in common installation locations. Often this
          is possible and allows the user to simply write:
<programlisting>
using yfc ;
</programlisting>
          </para></listitem>

        <listitem><para>Consider using facilities in the
          <code>tools/common</code> module. You can take a look at how
          <code>tools/gcc.jam</code> uses that module in the <code>init</code> rule.
          </para></listitem>

      </itemizedlist>
    </para>


      

  </section>

  </chapter>

<!--
     Local Variables:
     mode: xml
     sgml-indent-data: t     
     sgml-parent-document: ("userman.xml" "chapter")
     sgml-set-face: t
     End:
-->