Building Hybrid Systems with Boost.Python
-| Author: | -David Abrahams |
|---|---|
| Contact: | -dave@boost-consulting.com |
| Organization: | -Boost Consulting |
| Date: | -2003-03-19 |
| Author: | -Ralf W. Grosse-Kunstleve |
| Copyright: | -Copyright David Abrahams and Ralf W. Grosse-Kunstleve 2003. All rights reserved |
Abstract
-Boost.Python is an open source C++ library which provides a concise -IDL-like interface for binding C++ classes and functions to -Python. Leveraging the full power of C++ compile-time introspection -and of recently developed metaprogramming techniques, this is achieved -entirely in pure C++, without introducing a new syntax. -Boost.Python's rich set of features and high-level interface make it -possible to engineer packages from the ground up as hybrid systems, -giving programmers easy and coherent access to both the efficient -compile-time polymorphism of C++ and the extremely convenient run-time -polymorphism of Python.
-Introduction
-Python and C++ are in many ways as different as two languages could -be: while C++ is usually compiled to machine-code, Python is -interpreted. Python's dynamic type system is often cited as the -foundation of its flexibility, while in C++ static typing is the -cornerstone of its efficiency. C++ has an intricate and difficult -compile-time meta-language, while in Python, practically everything -happens at runtime.
-Yet for many programmers, these very differences mean that Python and -C++ complement one another perfectly. Performance bottlenecks in -Python programs can be rewritten in C++ for maximal speed, and -authors of powerful C++ libraries choose Python as a middleware -language for its flexible system integration capabilities. -Furthermore, the surface differences mask some strong similarities:
--
-
- 'C'-family control structures (if, while, for...) -
- Support for object-orientation, functional programming, and generic -programming (these are both multi-paradigm programming languages.) -
- Comprehensive operator overloading facilities, recognizing the -importance of syntactic variability for readability and -expressivity. -
- High-level concepts such as collections and iterators. -
- High-level encapsulation facilities (C++: namespaces, Python: modules) -to support the design of re-usable libraries. -
- Exception-handling for effective management of error conditions. -
- C++ idioms in common use, such as handle/body classes and -reference-counted smart pointers mirror Python reference semantics. -
Given Python's rich 'C' interoperability API, it should in principle -be possible to expose C++ type and function interfaces to Python with -an analogous interface to their C++ counterparts. However, the -facilities provided by Python alone for integration with C++ are -relatively meager. Compared to C++ and Python, 'C' has only very -rudimentary abstraction facilities, and support for exception-handling -is completely missing. 'C' extension module writers are required to -manually manage Python reference counts, which is both annoyingly -tedious and extremely error-prone. Traditional extension modules also -tend to contain a great deal of boilerplate code repetition which -makes them difficult to maintain, especially when wrapping an evolving -API.
-These limitations have lead to the development of a variety of wrapping -systems. SWIG is probably the most popular package for the -integration of C/C++ and Python. A more recent development is SIP, -which was specifically designed for interfacing Python with the Qt -graphical user interface library. Both SWIG and SIP introduce their -own specialized languages for customizing inter-language bindings. -This has certain advantages, but having to deal with three different -languages (Python, C/C++ and the interface language) also introduces -practical and mental difficulties. The CXX package demonstrates an -interesting alternative. It shows that at least some parts of -Python's 'C' API can be wrapped and presented through a much more -user-friendly C++ interface. However, unlike SWIG and SIP, CXX does -not include support for wrapping C++ classes as new Python types.
-The features and goals of Boost.Python overlap significantly with -many of these other systems. That said, Boost.Python attempts to -maximize convenience and flexibility without introducing a separate -wrapping language. Instead, it presents the user with a high-level -C++ interface for wrapping C++ classes and functions, managing much of -the complexity behind-the-scenes with static metaprogramming. -Boost.Python also goes beyond the scope of earlier systems by -providing:
--
-
- Support for C++ virtual functions that can be overridden in Python. -
- Comprehensive lifetime management facilities for low-level C++ -pointers and references. -
- Support for organizing extensions as Python packages, -with a central registry for inter-language type conversions. -
- A safe and convenient mechanism for tying into Python's powerful -serialization engine (pickle). -
- Coherence with the rules for handling C++ lvalues and rvalues that -can only come from a deep understanding of both the Python and C++ -type systems. -
The key insight that sparked the development of Boost.Python is that -much of the boilerplate code in traditional extension modules could be -eliminated using C++ compile-time introspection. Each argument of a -wrapped C++ function must be extracted from a Python object using a -procedure that depends on the argument type. Similarly the function's -return type determines how the return value will be converted from C++ -to Python. Of course argument and return types are part of each -function's type, and this is exactly the source from which -Boost.Python deduces most of the information required.
-This approach leads to user guided wrapping: as much information is -extracted directly from the source code to be wrapped as is possible -within the framework of pure C++, and some additional information is -supplied explicitly by the user. Mostly the guidance is mechanical -and little real intervention is required. Because the interface -specification is written in the same full-featured language as the -code being exposed, the user has unprecedented power available when -she does need to take control.
-Boost.Python Design Goals
-The primary goal of Boost.Python is to allow users to expose C++ -classes and functions to Python using nothing more than a C++ -compiler. In broad strokes, the user experience should be one of -directly manipulating C++ objects from Python.
-However, it's also important not to translate all interfaces too -literally: the idioms of each language must be respected. For -example, though C++ and Python both have an iterator concept, they are -expressed very differently. Boost.Python has to be able to bridge the -interface gap.
-It must be possible to insulate Python users from crashes resulting -from trivial misuses of C++ interfaces, such as accessing -already-deleted objects. By the same token the library should -insulate C++ users from low-level Python 'C' API, replacing -error-prone 'C' interfaces like manual reference-count management and -raw PyObject pointers with more-robust alternatives.
-Support for component-based development is crucial, so that C++ types -exposed in one extension module can be passed to functions exposed in -another without loss of crucial information like C++ inheritance -relationships.
-Finally, all wrapping must be non-intrusive, without modifying or -even seeing the original C++ source code. Existing C++ libraries have -to be wrappable by third parties who only have access to header files -and binaries.
-Hello Boost.Python World
-And now for a preview of Boost.Python, and how it improves on the raw -facilities offered by Python. Here's a function we might want to -expose:
-
-char const* greet(unsigned x)
-{
- static char const* const msgs[] = { "hello", "Boost.Python", "world!" };
-
- if (x > 2)
- throw std::range_error("greet: index out of range");
-
- return msgs[x];
-}
-
-To wrap this function in standard C++ using the Python 'C' API, we'd -need something like this:
-
-extern "C" // all Python interactions use 'C' linkage and calling convention
-{
- // Wrapper to handle argument/result conversion and checking
- PyObject* greet_wrap(PyObject* args, PyObject * keywords)
- {
- int x;
- if (PyArg_ParseTuple(args, "i", &x)) // extract/check arguments
- {
- char const* result = greet(x); // invoke wrapped function
- return PyString_FromString(result); // convert result to Python
- }
- return 0; // error occurred
- }
-
- // Table of wrapped functions to be exposed by the module
- static PyMethodDef methods[] = {
- { "greet", greet_wrap, METH_VARARGS, "return one of 3 parts of a greeting" }
- , { NULL, NULL, 0, NULL } // sentinel
- };
-
- // module initialization function
- DL_EXPORT init_hello()
- {
- (void) Py_InitModule("hello", methods); // add the methods to the module
- }
-}
-
-Now here's the wrapping code we'd use to expose it with Boost.Python:
-
-#include <boost/python.hpp>
-using namespace boost::python;
-BOOST_PYTHON_MODULE(hello)
-{
- def("greet", greet, "return one of 3 parts of a greeting");
-}
-
-and here it is in action:
-->>> import hello ->>> for x in range(3): -... print hello.greet(x) -... -hello -Boost.Python -world! --
Aside from the fact that the 'C' API version is much more verbose, -it's worth noting a few things that it doesn't handle correctly:
--
-
- The original function accepts an unsigned integer, and the Python -'C' API only gives us a way of extracting signed integers. The -Boost.Python version will raise a Python exception if we try to pass -a negative number to hello.greet, but the other one will proceed -to do whatever the C++ implementation does when converting an -negative integer to unsigned (usually wrapping to some very large -number), and pass the incorrect translation on to the wrapped -function. -
- That brings us to the second problem: if the C++ greet() -function is called with a number greater than 2, it will throw an -exception. Typically, if a C++ exception propagates across the -boundary with code generated by a 'C' compiler, it will cause a -crash. As you can see in the first version, there's no C++ -scaffolding there to prevent this from happening. Functions wrapped -by Boost.Python automatically include an exception-handling layer -which protects Python users by translating unhandled C++ exceptions -into a corresponding Python exception. -
- A slightly more-subtle limitation is that the argument conversion -used in the Python 'C' API case can only get that integer x in -one way. PyArg_ParseTuple can't convert Python long objects -(arbitrary-precision integers) which happen to fit in an unsigned -int but not in a signed long, nor will it ever handle a -wrapped C++ class with a user-defined implicit operator unsigned -int() conversion. Boost.Python's dynamic type conversion -registry allows users to add arbitrary conversion methods. -
Library Overview
-This section outlines some of the library's major features. Except as -neccessary to avoid confusion, details of library implementation are -omitted.
-Exposing Classes
-C++ classes and structs are exposed with a similarly-terse interface. -Given:
-
-struct World
-{
- void set(std::string msg) { this->msg = msg; }
- std::string greet() { return msg; }
- std::string msg;
-};
-
-The following code will expose it in our extension module:
-
-#include <boost/python.hpp>
-BOOST_PYTHON_MODULE(hello)
-{
- class_<World>("World")
- .def("greet", &World::greet)
- .def("set", &World::set)
- ;
-}
-
-Although this code has a certain pythonic familiarity, people -sometimes find the syntax bit confusing because it doesn't look like -most of the C++ code they're used to. All the same, this is just -standard C++. Because of their flexible syntax and operator -overloading, C++ and Python are great for defining domain-specific -(sub)languages -(DSLs), and that's what we've done in Boost.Python. To break it down:
-
-class_<World>("World")
-
-constructs an unnamed object of type class_<World> and passes -"World" to its constructor. This creates a new-style Python class -called World in the extension module, and associates it with the -C++ type World in the Boost.Python type conversion registry. We -might have also written:
-
-class_<World> w("World");
-
-but that would've been more verbose, since we'd have to name w -again to invoke its def() member function:
-
-w.def("greet", &World::greet)
-
-There's nothing special about the location of the dot for member -access in the original example: C++ allows any amount of whitespace on -either side of a token, and placing the dot at the beginning of each -line allows us to chain as many successive calls to member functions -as we like with a uniform syntax. The other key fact that allows -chaining is that class_<> member functions all return a reference -to *this.
-So the example is equivalent to:
-
-class_<World> w("World");
-w.def("greet", &World::greet);
-w.def("set", &World::set);
-
-It's occasionally useful to be able to break down the components of a -Boost.Python class wrapper in this way, but the rest of this article -will stick to the terse syntax.
-For completeness, here's the wrapped class in use:
-
->>> import hello
->>> planet = hello.World()
->>> planet.set('howdy')
->>> planet.greet()
-'howdy'
-
-Constructors
-Since our World class is just a plain struct, it has an -implicit no-argument (nullary) constructor. Boost.Python exposes the -nullary constructor by default, which is why we were able to write:
-->>> planet = hello.World() --
However, well-designed classes in any language may require constructor -arguments in order to establish their invariants. Unlike Python, -where __init__ is just a specially-named method, In C++ -constructors cannot be handled like ordinary member functions. In -particular, we can't take their address: &World::World is an -error. The library provides a different interface for specifying -constructors. Given:
-
-struct World
-{
- World(std::string msg); // added constructor
- ...
-
-we can modify our wrapping code as follows:
-
-class_<World>("World", init<std::string>())
- ...
-
-of course, a C++ class may have additional constructors, and we can -expose those as well by passing more instances of init<...> to -def():
-
-class_<World>("World", init<std::string>())
- .def(init<double, double>())
- ...
-
-Boost.Python allows wrapped functions, member functions, and -constructors to be overloaded to mirror C++ overloading.
-Data Members and Properties
-Any publicly-accessible data members in a C++ class can be easily -exposed as either readonly or readwrite attributes:
-
-class_<World>("World", init<std::string>())
- .def_readonly("msg", &World::msg)
- ...
-
-and can be used directly in Python:
-
->>> planet = hello.World('howdy')
->>> planet.msg
-'howdy'
-
-This does not result in adding attributes to the World instance -__dict__, which can result in substantial memory savings when -wrapping large data structures. In fact, no instance __dict__ -will be created at all unless attributes are explicitly added from -Python. Boost.Python owes this capability to the new Python 2.2 type -system, in particular the descriptor interface and property type.
-In C++, publicly-accessible data members are considered a sign of poor -design because they break encapsulation, and style guides usually -dictate the use of "getter" and "setter" functions instead. In -Python, however, __getattr__, __setattr__, and since 2.2, -property mean that attribute access is just one more -well-encapsulated syntactic tool at the programmer's disposal. -Boost.Python bridges this idiomatic gap by making Python property -creation directly available to users. If msg were private, we -could still expose it as attribute in Python as follows:
-
-class_<World>("World", init<std::string>())
- .add_property("msg", &World::greet, &World::set)
- ...
-
-The example above mirrors the familiar usage of properties in Python -2.2+:
-->>> class World(object): -... __init__(self, msg): -... self.__msg = msg -... def greet(self): -... return self.__msg -... def set(self, msg): -... self.__msg = msg -... msg = property(greet, set) --
Operator Overloading
-The ability to write arithmetic operators for user-defined types has -been a major factor in the success of both languages for numerical -computation, and the success of packages like NumPy attests to the -power of exposing operators in extension modules. Boost.Python -provides a concise mechanism for wrapping operator overloads. The -example below shows a fragment from a wrapper for the Boost rational -number library:
-
-class_<rational<int> >("rational_int")
- .def(init<int, int>()) // constructor, e.g. rational_int(3,4)
- .def("numerator", &rational<int>::numerator)
- .def("denominator", &rational<int>::denominator)
- .def(-self) // __neg__ (unary minus)
- .def(self + self) // __add__ (homogeneous)
- .def(self * self) // __mul__
- .def(self + int()) // __add__ (heterogenous)
- .def(int() + self) // __radd__
- ...
-
-The magic is performed using a simplified application of "expression -templates" [VELD1995], a technique originally developed for -optimization of high-performance matrix algebra expressions. The -essence is that instead of performing the computation immediately, -operators are overloaded to construct a type representing the -computation. In matrix algebra, dramatic optimizations are often -available when the structure of an entire expression can be taken into -account, rather than evaluating each operation "greedily". -Boost.Python uses the same technique to build an appropriate Python -method object based on expressions involving self.
-Inheritance
-C++ inheritance relationships can be represented to Boost.Python by adding -an optional bases<...> argument to the class_<...> template -parameter list as follows:
-
-class_<Derived, bases<Base1,Base2> >("Derived")
- ...
-
-This has two effects:
--
-
- When the class_<...> is created, Python type objects -corresponding to Base1 and Base2 are looked up in -Boost.Python's registry, and are used as bases for the new Python -Derived type object, so methods exposed for the Python Base1 -and Base2 types are automatically members of the Derived -type. Because the registry is global, this works correctly even if -Derived is exposed in a different module from either of its -bases. -
- C++ conversions from Derived to its bases are added to the -Boost.Python registry. Thus wrapped C++ methods expecting (a -pointer or reference to) an object of either base type can be -called with an object wrapping a Derived instance. Wrapped -member functions of class T are treated as though they have an -implicit first argument of T&, so these conversions are -neccessary to allow the base class methods to be called for derived -objects. -
Of course it's possible to derive new Python classes from wrapped C++ -class instances. Because Boost.Python uses the new-style class -system, that works very much as for the Python built-in types. There -is one significant detail in which it differs: the built-in types -generally establish their invariants in their __new__ function, so -that derived classes do not need to call __init__ on the base -class before invoking its methods :
-->>> class L(list): -... def __init__(self): -... pass -... ->>> L().reverse() ->>> --
Because C++ object construction is a one-step operation, C++ instance -data cannot be constructed until the arguments are available, in the -__init__ function:
-->>> class D(SomeBoostPythonClass): -... def __init__(self): -... pass -... ->>> D().some_boost_python_method() -Traceback (most recent call last): - File "<stdin>", line 1, in ? -TypeError: bad argument type for built-in operation --
This happened because Boost.Python couldn't find instance data of type -SomeBoostPythonClass within the D instance; D's __init__ -function masked construction of the base class. It could be corrected -by either removing D's __init__ function or having it call -SomeBoostPythonClass.__init__(...) explicitly.
-Virtual Functions
-Deriving new types in Python from extension classes is not very -interesting unless they can be used polymorphically from C++. In -other words, Python method implementations should appear to override -the implementation of C++ virtual functions when called through base -class pointers/references from C++. Since the only way to alter the -behavior of a virtual function is to override it in a derived class, -the user must build a special derived class to dispatch a polymorphic -class' virtual functions:
-
-//
-// interface to wrap:
-//
-class Base
-{
- public:
- virtual int f(std::string x) { return 42; }
- virtual ~Base();
-};
-
-int calls_f(Base const& b, std::string x) { return b.f(x); }
-
-//
-// Wrapping Code
-//
-
-// Dispatcher class
-struct BaseWrap : Base
-{
- // Store a pointer to the Python object
- BaseWrap(PyObject* self_) : self(self_) {}
- PyObject* self;
-
- // Default implementation, for when f is not overridden
- int f_default(std::string x) { return this->Base::f(x); }
- // Dispatch implementation
- int f(std::string x) { return call_method<int>(self, "f", x); }
-};
-
-...
- def("calls_f", calls_f);
- class_<Base, BaseWrap>("Base")
- .def("f", &Base::f, &BaseWrap::f_default)
- ;
-
-Now here's some Python code which demonstrates:
-->>> class Derived(Base): -... def f(self, s): -... return len(s) -... ->>> calls_f(Base(), 'foo') -42 ->>> calls_f(Derived(), 'forty-two') -9 --
Things to notice about the dispatcher class:
--
-
- The key element which allows overriding in Python is the -call_method invocation, which uses the same global type -conversion registry as the C++ function wrapping does to convert its -arguments from C++ to Python and its return type from Python to C++. -
- Any constructor signatures you wish to wrap must be replicated with -an initial PyObject* argument -
- The dispatcher must store this argument so that it can be used to -invoke call_method -
- The f_default member function is needed when the function being -exposed is not pure virtual; there's no other way Base::f can be -called on an object of type BaseWrap, since it overrides f. -
Deeper Reflection on the Horizon?
-Admittedly, this formula is tedious to repeat, especially on a project -with many polymorphic classes. That it is neccessary reflects some -limitations in C++'s compile-time introspection capabilities: there's -no way to enumerate the members of a class and find out which are -virtual functions. At least one very promising project has been -started to write a front-end which can generate these dispatchers (and -other wrapping code) automatically from C++ headers.
-Pyste is being developed by Bruno da Silva de Oliveira. It builds on -GCC_XML, which generates an XML version of GCC's internal program -representation. Since GCC is a highly-conformant C++ compiler, this -ensures correct handling of the most-sophisticated template code and -full access to the underlying type system. In keeping with the -Boost.Python philosophy, a Pyste interface description is neither -intrusive on the code being wrapped, nor expressed in some unfamiliar -language: instead it is a 100% pure Python script. If Pyste is -successful it will mark a move away from wrapping everything directly -in C++ for many of our users. It will also allow us the choice to -shift some of the metaprogram code from C++ to Python. We expect that -soon, not only our users but the Boost.Python developers themselves -will be "thinking hybrid" about their own code.
-Serialization
-Serialization is the process of converting objects in memory to a -form that can be stored on disk or sent over a network connection. The -serialized object (most often a plain string) can be retrieved and -converted back to the original object. A good serialization system will -automatically convert entire object hierarchies. Python's standard -pickle module is just such a system. It leverages the language's strong -runtime introspection facilities for serializing practically arbitrary -user-defined objects. With a few simple and unintrusive provisions this -powerful machinery can be extended to also work for wrapped C++ objects. -Here is an example:
-
-#include <string>
-
-struct World
-{
- World(std::string a_msg) : msg(a_msg) {}
- std::string greet() const { return msg; }
- std::string msg;
-};
-
-#include <boost/python.hpp>
-using namespace boost::python;
-
-struct World_picklers : pickle_suite
-{
- static tuple
- getinitargs(World const& w) { return make_tuple(w.greet()); }
-};
-
-BOOST_PYTHON_MODULE(hello)
-{
- class_<World>("World", init<std::string>())
- .def("greet", &World::greet)
- .def_pickle(World_picklers())
- ;
-}
-
-Now let's create a World object and put it to rest on disk:
-
->>> import hello
->>> import pickle
->>> a_world = hello.World("howdy")
->>> pickle.dump(a_world, open("my_world", "w"))
-
-In a potentially different script on a potentially different -computer with a potentially different operating system:
-
->>> import pickle
->>> resurrected_world = pickle.load(open("my_world", "r"))
->>> resurrected_world.greet()
-'howdy'
-
-Of course the cPickle module can also be used for faster -processing.
-Boost.Python's pickle_suite fully supports the pickle protocol -defined in the standard Python documentation. Like a __getinitargs__ -function in Python, the pickle_suite's getinitargs() is responsible for -creating the argument tuple that will be use to reconstruct the pickled -object. The other elements of the Python pickling protocol, -__getstate__ and __setstate__ can be optionally provided via C++ -getstate and setstate functions. C++'s static type system allows the -library to ensure at compile-time that nonsensical combinations of -functions (e.g. getstate without setstate) are not used.
-Enabling serialization of more complex C++ objects requires a little -more work than is shown in the example above. Fortunately the -object interface (see next section) greatly helps in keeping the -code manageable.
-Object interface
-Experienced 'C' language extension module authors will be familiar -with the ubiquitous PyObject*, manual reference-counting, and the -need to remember which API calls return "new" (owned) references or -"borrowed" (raw) references. These constraints are not just -cumbersome but also a major source of errors, especially in the -presence of exceptions.
-Boost.Python provides a class object which automates reference -counting and provides conversion to Python from C++ objects of -arbitrary type. This significantly reduces the learning effort for -prospective extension module writers.
-Creating an object from any other type is extremely simple:
-
-object s("hello, world"); // s manages a Python string
-
-object has templated interactions with all other types, with -automatic to-python conversions. It happens so naturally that it's -easily overlooked:
--object ten_Os = 10 * s[4]; // -> "oooooooooo" --
In the example above, 4 and 10 are converted to Python objects -before the indexing and multiplication operations are invoked.
-The extract<T> class template can be used to convert Python objects -to C++ types:
--double x = extract<double>(o); --
If a conversion in either direction cannot be performed, an -appropriate exception is thrown at runtime.
-The object type is accompanied by a set of derived types -that mirror the Python built-in types such as list, dict, -tuple, etc. as much as possible. This enables convenient -manipulation of these high-level types from C++:
--dict d; -d["some"] = "thing"; -d["lucky_number"] = 13; -list l = d.keys(); --
This almost looks and works like regular Python code, but it is pure -C++. Of course we can wrap C++ functions which accept or return -object instances.
-Thinking hybrid
-Because of the practical and mental difficulties of combining -programming languages, it is common to settle a single language at the -outset of any development effort. For many applications, performance -considerations dictate the use of a compiled language for the core -algorithms. Unfortunately, due to the complexity of the static type -system, the price we pay for runtime performance is often a -significant increase in development time. Experience shows that -writing maintainable C++ code usually takes longer and requires far -more hard-earned working experience than developing comparable Python -code. Even when developers are comfortable working exclusively in -compiled languages, they often augment their systems by some type of -ad hoc scripting layer for the benefit of their users without ever -availing themselves of the same advantages.
-Boost.Python enables us to think hybrid. Python can be used for -rapidly prototyping a new application; its ease of use and the large -pool of standard libraries give us a head start on the way to a -working system. If necessary, the working code can be used to -discover rate-limiting hotspots. To maximize performance these can -be reimplemented in C++, together with the Boost.Python bindings -needed to tie them back into the existing higher-level procedure.
-Of course, this top-down approach is less attractive if it is clear -from the start that many algorithms will eventually have to be -implemented in C++. Fortunately Boost.Python also enables us to -pursue a bottom-up approach. We have used this approach very -successfully in the development of a toolbox for scientific -applications. The toolbox started out mainly as a library of C++ -classes with Boost.Python bindings, and for a while the growth was -mainly concentrated on the C++ parts. However, as the toolbox is -becoming more complete, more and more newly added functionality can be -implemented in Python.
-
This figure shows the estimated ratio of newly added C++ and Python -code over time as new algorithms are implemented. We expect this -ratio to level out near 70% Python. Being able to solve new problems -mostly in Python rather than a more difficult statically typed -language is the return on our investment in Boost.Python. The ability -to access all of our code from Python allows a broader group of -developers to use it in the rapid development of new applications.
-Development history
-The first version of Boost.Python was developed in 2000 by Dave -Abrahams at Dragon Systems, where he was privileged to have Tim Peters -as a guide to "The Zen of Python". One of Dave's jobs was to develop -a Python-based natural language processing system. Since it was -eventually going to be targeting embedded hardware, it was always -assumed that the compute-intensive core would be rewritten in C++ to -optimize speed and memory footprint 1. The project also wanted to -test all of its C++ code using Python test scripts 2. The only -tool we knew of for binding C++ and Python was SWIG, and at the time -its handling of C++ was weak. It would be false to claim any deep -insight into the possible advantages of Boost.Python's approach at -this point. Dave's interest and expertise in fancy C++ template -tricks had just reached the point where he could do some real damage, -and Boost.Python emerged as it did because it filled a need and -because it seemed like a cool thing to try.
-This early version was aimed at many of the same basic goals we've -described in this paper, differing most-noticeably by having a -slightly more cumbersome syntax and by lack of special support for -operator overloading, pickling, and component-based development. -These last three features were quickly added by Ullrich Koethe and -Ralf Grosse-Kunstleve 3, and other enthusiastic contributors arrived -on the scene to contribute enhancements like support for nested -modules and static member functions.
-By early 2001 development had stabilized and few new features were -being added, however a disturbing new fact came to light: Ralf had -begun testing Boost.Python on pre-release versions of a compiler using -the EDG front-end, and the mechanism at the core of Boost.Python -responsible for handling conversions between Python and C++ types was -failing to compile. As it turned out, we had been exploiting a very -common bug in the implementation of all the C++ compilers we had -tested. We knew that as C++ compilers rapidly became more -standards-compliant, the library would begin failing on more -platforms. Unfortunately, because the mechanism was so central to the -functioning of the library, fixing the problem looked very difficult.
-Fortunately, later that year Lawrence Berkeley and later Lawrence -Livermore National labs contracted with Boost Consulting for support -and development of Boost.Python, and there was a new opportunity to -address fundamental issues and ensure a future for the library. A -redesign effort began with the low level type conversion architecture, -building in standards-compliance and support for component-based -development (in contrast to version 1 where conversions had to be -explicitly imported and exported across module boundaries). A new -analysis of the relationship between the Python and C++ objects was -done, resulting in more intuitive handling for C++ lvalues and -rvalues.
-The emergence of a powerful new type system in Python 2.2 made the -choice of whether to maintain compatibility with Python 1.5.2 easy: -the opportunity to throw away a great deal of elaborate code for -emulating classic Python classes alone was too good to pass up. In -addition, Python iterators and descriptors provided crucial and -elegant tools for representing similar C++ constructs. The -development of the generalized object interface allowed us to -further shield C++ programmers from the dangers and syntactic burdens -of the Python 'C' API. A great number of other features including C++ -exception translation, improved support for overloaded functions, and -most significantly, CallPolicies for handling pointers and -references, were added during this period.
-In October 2002, version 2 of Boost.Python was released. Development -since then has concentrated on improved support for C++ runtime -polymorphism and smart pointers. Peter Dimov's ingenious -boost::shared_ptr design in particular has allowed us to give the -hybrid developer a consistent interface for moving objects back and -forth across the language barrier without loss of information. At -first, we were concerned that the sophistication and complexity of the -Boost.Python v2 implementation might discourage contributors, but the -emergence of Pyste and several other significant feature -contributions have laid those fears to rest. Daily questions on the -Python C++-sig and a backlog of desired improvements show that the -library is getting used. To us, the future looks bright.
-Conclusions
-Boost.Python achieves seamless interoperability between two rich and -complimentary language environments. Because it leverages template -metaprogramming to introspect about types and functions, the user -never has to learn a third syntax: the interface definitions are -written in concise and maintainable C++. Also, the wrapping system -doesn't have to parse C++ headers or represent the type system: the -compiler does that work for us.
-Computationally intensive tasks play to the strengths of C++ and are -often impossible to implement efficiently in pure Python, while jobs -like serialization that are trivial in Python can be very difficult in -pure C++. Given the luxury of building a hybrid software system from -the ground up, we can approach design with new confidence and power.
-Citations
-| [VELD1995] | T. Veldhuizen, "Expression Templates," C++ Report, -Vol. 7 No. 5 June 1995, pp. 26-31. -http://osl.iu.edu/~tveldhui/papers/Expression-Templates/exprtmpl.html |
Footnotes
-| [1] | In retrospect, it seems that "thinking hybrid" from the -ground up might have been better for the NLP system: the -natural component boundaries defined by the pure python -prototype turned out to be inappropriate for getting the -desired performance and memory footprint out of the C++ core, -which eventually caused some redesign overhead on the Python -side when the core was moved to C++. |
| [2] | We also have some reservations about driving all C++ -testing through a Python interface, unless that's the only way -it will be ultimately used. Any transition across language -boundaries with such different object models can inevitably -mask bugs. |
| [3] | These features were expressed very differently in v1 of -Boost.Python |