python/doc/building.rst

.. Copyright David Abrahams 2006. Distributed under the Boost
.. Software License, Version 1.0. (See accompanying
.. file LICENSE_1_0.txt or copy at
.. http://www.boost.org/LICENSE_1_0.txt)

==============================================
 |(logo)|__ Boost.Python Build and Test HOWTO
==============================================

.. |(logo)| image:: ../../../boost.png
   :alt: Boost C++ Libraries:
   :class: boost-logo

__ ../index.html


.. section-numbering::
   :depth: 2

.. contents:: Contents
   :depth: 2
   :class: sidebar small

.. |newer| replace:: *newer*

Requirements
============

Boost.Python requires `Python 2.2`_ [#2.2]_ *or* |newer|__.  

.. _Python 2.2: http://www.python.org/2.2
__ http://www.python.org

Background
==========

There are two basic models for combining C++ and Python:

- extending_, in which the end-user launches the Python interpreter
  executable and imports Python “extension modules” written in C++.
  Think of taking a library written in C++ and giving it a Python
  interface so Python programmers can use it.  From Python, these
  modules look just like regular Python modules.

- embedding_, in which the end-user launches a program written
  in C++ that in turn invokes the Python interpreter as a library
  subroutine.  Think of adding scriptability to an existing
  application.

.. _extending: http://www.python.org/doc/current/ext/intro.html
.. _embedding: http://www.python.org/doc/current/ext/embedding.html

The key distinction between extending and embedding is the location
of the C++ ``main()`` function: in the Python interpreter executable,
or in some other program, respectively.  Note that even when
embedding Python in another program, `extension modules are often
the best way to make C/C++ functionality accessible to Python
code`__, so the use of extension modules is really at the heart of
both models.

__ http://www.python.org/doc/current/ext/extending-with-embedding.html

Except in rare cases, extension modules are built as
dynamically-loaded libraries with a single entry point, which means
you can change them without rebuilding either the other extension
modules or the executable containing ``main()``.

.. _quickstart:

No-Install Quickstart
=====================

There is no need to “install Boost” in order to get started using
Boost.Python.  These instructions use Boost.Build_ projects,
which will build those binaries as soon as they're needed.  Your
first tests may take a little longer while you wait for
Boost.Python to build, but doing things this way will save you from
worrying about build intricacies like which library binaries to use
for a specific compiler configuration and figuring out the right
compiler options to use yourself.

.. .. raw:: html

   <div style="width:50%">

.. Note:: Of course it's possible to use other build systems to
   build Boost.Python and its extensions, but they are not
   officially supported by Boost.  Moreover **99% of all “I can't
   build Boost.Python” problems come from trying to use another
   build system** without first following these instructions.

   If you want to use another system anyway, we suggest that you
   follow these instructions, and then invoke ``bjam`` with the

   .. parsed-literal::

     ``-a -o``\ *filename* 

   options to dump the build commands it executes to a file, so
   you can see what your alternate build system needs to do.

.. .. raw:: html
 
   </div>

.. _Boost.Build: ../../../tools/build/index.html

Basic Procedure
---------------

1. Get Boost; see sections 1 and 2 [`Unix/Linux`__, `Windows`__\ ] of the
   Boost `Getting Started Guide`_.

   __ ../../../more/getting_started/unix-variants.html#get-boost
   __ ../../../more/getting_started/windows.html#get-boost

2. Get the ``bjam`` build driver.  See section 5 [`Unix/Linux`__,
   `Windows`__\ ] of the Boost `Getting Started Guide`_.

   __ ../../../more/getting_started/unix-variants.html#prepare-to-use-a-boost-library-binary
   __ ../../../more/getting_started/windows.html#prepare-to-use-a-boost-library-binary


3. cd into the ``libs/python/example/quickstart/`` directory of your
   Boost installation, which contains a small example project.

4. Invoke ``bjam``.  Replace the “\ ``stage``\ “ argument from the
   example invocation from section 5 of the `Getting Started
   Guide`_ with “\ ``test``\ ,“ to build all the test targets.  Also add
   the argument “\ ``--verbose-test``\ ” to see the output generated by
   the tests when they are run.

   On Windows, your ``bjam`` invocation might look something like:

   .. parsed-literal::

     C:\\boost_1_34_0\\…\\quickstart> **bjam toolset=msvc --verbose-test test**

   and on Unix variants, perhaps,

   .. parsed-literal::

     ~/boost_1_34_0/…/quickstart$ **bjam toolset=gcc --verbose-test test**

.. Admonition:: Note to Windows Users

   For the sake of concision, the rest of this guide will use
   unix-style forward slashes in pathnames instead of the
   backslashes with which you may be more familiar.  The forward
   slashes should work everywhere except in `Command Prompt`_
   windows, where you should use backslashes.

   .. _Command Prompt: ../../../more/getting_started/windows.html#command-prompt

If you followed this procedure successfully, you will have built an
extension module called ``extending`` and tested it by running a
Python script called ``test_extending.py``.  You will also have
built and run a simple application called ``embedding`` that embeds
python.

.. _Getting Started Guide: ../../../more/getting_started/index.html

In Case of Trouble
------------------

If you're seeing lots of compiler and/or linker error messages,
it's probably because Boost.Build is having trouble finding your
Python installation.  You might want to pass the
``--debug-configuration`` option to ``bjam`` the first few times
you invoke it, to make sure that Boost.Build is correctly locating
all the parts of your Python installation.  If it isn't, consider
`Configuring Boost.Build`_ as detailed below.

If you're still having trouble, Someone on one of the following
mailing lists may be able to help:

* The `Boost.Build mailing list`__ for issues related to Boost.Build
* The Python `C++ Sig`__ for issues specifically related to Boost.Python

__ http://www.boost.org/more/mailing_lists.htm#jamboost
__ http://www.boost.org/more/mailing_lists.htm#cplussig

In Case Everything Seemed to Work
---------------------------------

Rejoice!  If you're new to Boost.Python, at this point it might be
a good idea to ignore build issues for a while and concentrate on
learning the library by going through the tutorial_ and perhaps
some of the `reference documentation`_, trying out what you've
learned about the API by modifying the quickstart project.

.. _reference documentation: v2/reference.html
.. _tutorial: tutorial/index.html

Modifying the Example Project
-----------------------------

If you're content to keep your extension module forever in one
source file called |extending.cpp|_, inside your Boost
distribution, and import it forever as ``extending``, then you can
stop here.  However, it's likely that you will want to make a few
changes.  There are a few things you can do without having to learn
Boost.Build_ in depth.

The project you just built is specified in two files in the current
directory: |boost-build.jam|_, which tells ``bjam`` where it can
find the interpreted code of the Boost build system, and
|Jamroot|_, which describes the targets you just built.  These
files are heavily commented, so they should be easy to modify.
Take care, however, to preserve whitespace.  Punctuation such as
``;`` will not be recognized as intended by ``bjam`` if it is not
surrounded by whitespace.

.. |boost-build.jam| replace:: ``boost-build.jam``
.. _boost-build.jam: ../example/quickstart/boost-build.jam

.. |Jamroot| replace:: ``Jamroot``
.. _Jamroot: ../example/quickstart/Jamroot

.. |extending.cpp| replace:: ``extending.cpp``
.. _extending.cpp: ../example/quickstart/extending.cpp

Relocate the Project
....................

You'll probably want to copy this project elsewhere so you can
change it without modifying your Boost distribution.  To do that,
simply

a. copy the entire ``libs/python/example/quickstart/`` directory
   into a new directory.

b. In the new copies of |boost-build.jam|_ and |Jamroot|_, locate
   the relative path near the top of the file that is clearly
   marked by a comment, and edit that path so that it refers to the
   same directory your Boost distribution as it referred to when
   the file was in its original location in the
   ``libs/python/example/quickstart/`` directory.

For example, if you moved the project from
``/home/dave/boost_1_34_0/libs/python/example/quickstart`` to
``/home/dave/my-project``, you could change the first path in
|boost-build.jam|_ from

.. parsed-literal::

  **../../../..**\ /tools/build/v2

to

.. parsed-literal::

  **/home/dave/boost_1_34_0**\ /tools/build/v2

and change the first path in |Jamroot|_ from

.. parsed-literal::

  **../../../..**

to

.. parsed-literal::

  **/home/dave/boost_1_34_0**

Add New or Change Names of Existing Source Files
................................................

The names of additional source files involved in building your
extension module or embedding application can be listed in
|Jamroot|_ right alongside ``extending.cpp`` or ``embedding.cpp``
respectively.  Just be sure to leave whitespace around each
filename::

  … file1.cpp file2.cpp file3.cpp …

Naturally, if you want to change the name of a source file you can
tell Boost.Build about it by editing the name in |Jamroot|_.

Change the Name of your Extension Module
........................................

The name of the extension module is determined by two things:

1. the name in |Jamroot|_ immediately following ``python-extension``, and 
2. the name passed to ``BOOST_PYTHON_MODULE`` in |extending.cpp|_.

To change the name of the extension module from ``extending`` to
``hello``, you'd edit |Jamroot|_, changing

.. parsed-literal::

  python-extension **extending** : extending.cpp ;

to

.. parsed-literal::

  python-extension **hello** : extending.cpp ;

and you'd edit extending.cpp, changing

.. parsed-literal::

  BOOST_PYTHON_MODULE(\ **extending**\ )

to

.. parsed-literal::

  BOOST_PYTHON_MODULE(\ **hello**\ )

Installing Boost.Python on your System
======================================

Since Boost.Python is a separately-compiled (as opposed to
`header-only`_) library, its user relies on the services of a
Boost.Python library binary.  

.. _header-only: ../../../more/getting_started/windows.html#header-only-libraries

If you need a regular installation of the Boost.Python library
binaries on your system, the Boost `Getting Started Guide`_ will
walk you through the steps of creating one.  If building binaries
from source, you might want to supply the ``--with-python``
argument to ``bjam`` (or the ``--with-libraries=python`` argument
to ``configure``), so only the Boost.Python binary will be built,
rather than all the Boost binaries.


Configuring Boost.Build
=======================

As described in the `Boost.Build reference manual`__, a file called
``user-config.jam`` in your home directory [#home-dir]_ is used to
specify the tools and libraries available to the build system. You
may need to create or edit ``user-config.jam`` to tell Boost.Build
how to invoke Python, ``#include`` its headers, and link with its
libraries.

__ http://www.boost.orgdoc/html/bbv2/advanced.html#bbv2.advanced.configuration

.. Admonition:: Users of Unix-Variant OSes

   If you are using a unix-variant OS and you ran Boost's
   ``configure`` script, it may have generated a
   ``user-config.jam`` for you. [#overwrite]_ If your ``configure``\
   /\ ``make`` sequence was successful and Boost.Python binaries
   were built, your ``user-config.jam`` file is probably already
   correct.

If you have one fairly “standard” python installation for your
platform, you might not need to do anything special to describe it.  If
you haven't configured python in ``user-config.jam`` (and you don't
specify ``--without-python`` on the Boost.Build command line),
Boost.Build will automatically execute the equivalent of ::

  import toolset : using ; 
  using python ;

which automatically looks for Python in the most likely places.
However, that only happens when using the Boost.Python project file
(e.g. when referred to by another project as in the quickstart_
method).  If instead you are linking against separately-compiled
Boost.Python binaries, you should set up a ``user-config.jam`` file
with at least the minimal incantation above.

Python Configuration Parameters
-------------------------------

If you have several versions of Python installed, or Python is
installed in an unusual way, you may want to supply any or all of
the following optional parameters to ``using python``.

version
  the version of Python to use.  Should be in Major.Minor
  format, for example, ``2.3``.  Do not include the subminor
  version (i.e. *not* ``2.5.1``).  If you have multiple Python
  versions installed, the version will usually be the only
  configuration argument required.

cmd-or-prefix
  preferably, a command that invokes a Python interpreter.
  Alternatively, the installation prefix for Python libraries and
  header files.  Only use the alternative formulation if there is
  no appropriate Python executable available.

includes
  the ``#include`` paths for Python headers.  Normally the correct
  path(s) will be automatically deduced from ``version`` and/or
  ``cmd-or-prefix``.  

libraries
  the path to Python library binaries.  On MacOS/Darwin,
  you can also pass the path of the Python framework.  Normally the
  correct path(s) will be automatically deduced from ``version``
  and/or ``cmd-or-prefix``. 

condition
  if specified, should be a set of Boost.Build
  properties that are matched against the build configuration when
  Boost.Build selects a Python configuration to use.  See examples
  below for details.

extension-suffix
  A string to append to the name of extension
  modules before the true filename extension.  You almost certainly
  don't need to use this.  Usually this suffix is only used when
  targeting a Windows debug build of Python, and will be set
  automatically for you based on the value of the
  |python-debugging|_ feature.  However, at least one Linux
  distribution (Ubuntu Feisty Fawn) has a specially configured
  `python-dbg`__ package that claims to use such a suffix.

.. |python-debugging| replace:: ``<python-debugging>``

__ https://wiki.ubuntu.com/PyDbgBuilds


Examples
--------

Note that in the examples below, case and *especially whitespace* are
significant.

- If you have both python 2.5 and python 2.4 installed,
  ``user-config.jam`` might contain::
  
    using python : 2.5 ;  # Make both versions of Python available 

    using python : 2.4 ;  # To build with python 2.4, add python=2.4 
                          # to your command line.

  The first version configured (2.5) becomes the default.  To build
  against python 2.4, add ``python=2.4`` to the ``bjam`` command line.

- If you have python installed in an unusual location, you might
  supply the path to the interpreter in the ``cmd-or-prefix``
  parameter::

    using python : : /usr/local/python-2.6-beta/bin/python ;

- If you have a separate build of Python for use with a particular
  toolset, you might supply that toolset in the ``condition``
  parameter::

    using python ;  # use for most toolsets
    
    # Use with Intel C++ toolset
    using python 
         : # version
         : c:\\Devel\\Python-2.5-IntelBuild\\PCBuild\\python # cmd-or-prefix
         : # includes
         : # libraries
         : <toolset>intel # condition
         ;


- If you have downloaded the Python sources and built both the
  normal and the “\ `python debugging`_\ ” builds from source on
  Windows, you might see::

    using python : 2.5 : C:\\src\\Python-2.5\\PCBuild\\python ;
    using python : 2.5 : C:\\src\\Python-2.5\\PCBuild\\python_d
      : # includes
      : # libs
      : <python-debugging>on ;

- You can set up your user-config.jam so a bjam built under Windows 
  can build/test both Windows and Cygwin_ python extensions.  Just pass
  ``<target-os>cygwin`` in the ``condition`` parameter
  for the cygwin python installation::

    # windows installation
    using python ;

    # cygwin installation
    using python : : c:\\cygwin\\bin\\python2.5 : : : <target-os>cygwin ;

  when you put target-os=cygwin in your build request, it should build
  with the cygwin version of python: [#flavor]_

    bjam target-os=cygwin toolset=gcc 

  This is supposed to work the other way, too (targeting windows
  python with a Cygwin_ bjam) but it seems as though the support in
  Boost.Build's toolsets for building that way is broken at the
  time of this writing.

- Note that because of `the way Boost.Build currently selects target
  alternatives`__, you might have be very explicit in your build
  requests.  For example, given::

    using python : 2.5 ; # a regular windows build
    using python : 2.4 : : : : <target-os>cygwin ;

  building with ::

    bjam target-os=cygwin

  will yield an error.  Instead, you'll need to write::

    bjam target-os=cygwin/python=2.4

.. _Cygwin: http://cygwin.com

__ http://zigzag.cs.msu.su/boost.build/wiki/AlternativeSelection

Choosing a Boost.Python Library Binary
======================================

If—instead of letting Boost.Build construct and link with the right
libraries automatically—you choose to use a pre-built Boost.Python
library, you'll need to think about which one to link with.  The
Boost.Python binary comes in both static and dynamic flavors.  Take
care to choose the right flavor for your application. [#naming]_

The Dynamic Binary
------------------

The dynamic library is the safest and most-versatile choice:

- A single copy of the library code is used by all extension
  modules built with a given toolset. [#toolset-specific]_

- The library contains a type conversion registry.  Because one
  registry is shared among all extension modules, instances of a
  class exposed to Python in one dynamically-loaded extension
  module can be passed to functions exposed in another such module.

The Static Binary
-----------------

It might be appropriate to use the static Boost.Python library in
any of the following cases:

- You are extending_ python and the types exposed in your
  dynamically-loaded extension module don't need to be used by any
  other Boost.Python extension modules, and you don't care if the
  core library code is duplicated among them.

- You are embedding_ python in your application and either:

  - You are targeting a Unix variant OS other than MacOS or AIX,
    where the dynamically-loaded extension modules can “see” the
    Boost.Python library symbols that are part of the executable.

  - Or, you have statically linked some Boost.Python extension
    modules into your application and you don't care if any
    dynamically-loaded Boost.Python extension modules are able to
    use the types exposed by your statically-linked extension
    modules (and vice-versa).

``#include`` Issues
===================

1. If you should ever have occasion to ``#include "python.h"``
   directly in a translation unit of a program using Boost.Python,
   use ``#include "boost/python/detail/wrap_python.hpp"`` instead.
   It handles several issues necessary for use with Boost.Python,
   one of which is mentioned in the next section.

2. Be sure not to ``#include`` any system headers before
   ``wrap_python.hpp``.  This restriction is actually imposed by
   Python, or more properly, by Python's interaction with your
   operating system.  See
   http://docs.python.org/ext/simpleExample.html for details.

.. _python-debugging:
.. _python debugging:

Python Debugging Builds
=======================

Python can be built in a special “python debugging” configuration
that adds extra checks and instrumentation that can be very useful
for developers of extension modules.  The data structures used by
the debugging configuration contain additional members, so **a
Python executable built with python debugging enabled cannot be
used with an extension module or library compiled without it, and
vice-versa.**

Since pre-built “python debugging” versions of the Python
executable and libraries are not supplied with most distributions
of Python, [#get-debug-build]_ and we didn't want to force our users
to build them, Boost.Build does not automatically enable python
debugging in its ``debug`` build variant (which is the default).
Instead there is a special build property called
``python-debugging`` that, when used as a build property, will
define the right preprocessor symbols and select the right
libraries to link with.

On unix-variant platforms, the debugging versions of Python's data
structures will only be used if the symbol ``Py_DEBUG`` is defined.
On many windows compilers, when extension modules are built with
the preprocessor symbol ``_DEBUG``, Python defaults to force
linking with a special debugging version of the Python DLL.  Since
that symbol is very commonly used even when Python is not present,
Boost.Python temporarily undefines _DEBUG when Python.h
is #included from ``boost/python/detail/wrap_python.hpp`` - unless
``BOOST_DEBUG_PYTHON`` is defined.  The upshot is that if you want
“python debugging”and you aren't using Boost.Build, you should make
sure ``BOOST_DEBUG_PYTHON`` is defined, or python debugging will be
suppressed.

Testing Boost.Python
====================

To run the full test suite for Boost.Python, invoke ``bjam`` in the
``libs/python/test`` subdirectory of your Boost distribution.

Notes for MinGW (and Cygwin with -mno-cygwin) GCC Users
=======================================================

If you are using a version of Python prior to 2.4.1 with a MinGW
prior to 3.0.0 (with binutils-2.13.90-20030111-1), you will need to
create a MinGW-compatible version of the Python library; the one
shipped with Python will only work with a Microsoft-compatible
linker. Follow the instructions in the “Non-Microsoft” section of
the “Building Extensions: Tips And Tricks” chapter in `Installing
Python Modules`__ to create ``libpythonXX.a``, where ``XX``
corresponds to the major and minor version numbers of your Python
installation.

__ http://www.python.org/doc/current/inst/index.html

-----------------------------

.. [#2.2] Note that although we tested earlier versions of
   Boost.Python with Python 2.2, and we don't *think* we've done
   anything to break compatibility, this release of Boost.Python
   may not have been tested with versions of Python earlier than
   2.4, so we're not 100% sure that python 2.2 and 2.3 are
   supported.

.. [#naming] Information about how to identify the
   static and dynamic builds of Boost.Python:

   * `on Windows`__
   * `on Unix variants`__

   __ ../../../more/getting_started/windows.html#library-naming
   __ ../../../more/getting_started/unix-variants.html#library-naming

.. [#toolset-specific] Because of the way most \*nix platforms
   share symbols among dynamically-loaded objects, I'm not certain
   that extension modules built with different compiler toolsets
   will always use different copies of the Boost.Python library
   when loaded into the same Python instance.  Not using different
   libraries could be a good thing if the compilers have compatible
   ABIs, because extension modules built with the two libraries
   would be interoperable.  Otherwise, it could spell disaster,
   since an extension module and the Boost.Python library would
   have different ideas of such things as class layout. I would
   appreciate someone doing the experiment to find out what
   happens.

.. [#overwrite] ``configure`` overwrites the existing
   ``user-config.jam`` in your home directory
   (if any) after making a backup of the old version.

.. [#flavor] Note that the ``<target-os>cygwin`` feature is
   different from the ``<flavor>cygwin`` subfeature of the ``gcc``
   toolset, and you might need handle both explicitly if you also
   have a MinGW GCC installed.

.. [#home-dir] Windows users, your home directory can be
   found by typing::

     ECHO %HOMEDRIVE%%HOMEPATH%

   into a `command prompt`_ window.

.. [#get-debug-build] On Unix and similar platforms, a debugging
   python and associated libraries are built by adding
   ``--with-pydebug`` when configuring the Python build. On
   Windows, the debugging version of Python is generated by
   the "Win32 Debug" target of the Visual Studio project in the
   PCBuild subdirectory of a full Python source code distribution.