mirror of
https://github.com/boostorg/python.git
synced 2026-01-20 04:42:28 +00:00
337 lines
12 KiB
HTML
337 lines
12 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0//EN"
|
|
"http://www.w3.org/TR/REC-html40/strict.dtd">
|
|
|
|
<title>Cross-extension-module dependencies</title>
|
|
|
|
<div>
|
|
|
|
<img src="../../../c++boost.gif"
|
|
alt="c++boost.gif (8819 bytes)"
|
|
align="center"
|
|
width="277" height="86">
|
|
|
|
<hr>
|
|
<h1>Cross-extension-module dependencies</h1>
|
|
|
|
It is good programming practice to organize large projects as modules
|
|
that interact with each other via well defined interfaces. With
|
|
Boost.Python it is possible to reflect this organization at the C++
|
|
level at the Python level. This is, each logical C++ module can be
|
|
organized as a separate Python extension module.
|
|
|
|
<p>
|
|
At first sight this might seem natural and straightforward. However, it
|
|
is a fairly complex problem to establish cross-extension-module
|
|
dependencies while maintaining the same ease of use Boost.Python
|
|
provides for classes that are wrapped in the same extension module. To
|
|
a large extent this complexity can be hidden from the author of a
|
|
Boost.Python extension module, but not entirely.
|
|
|
|
<hr>
|
|
<h2>The recipe</h2>
|
|
|
|
Suppose there is an extension module that exposes certain instances of
|
|
the C++ <tt>std::vector</tt> template library such that it can be used
|
|
from Python in the following manner:
|
|
|
|
<pre>
|
|
import std_vector
|
|
v = std_vector.double([1, 2, 3, 4])
|
|
v.push_back(5)
|
|
v.size()
|
|
</pre>
|
|
|
|
Suppose the <tt>std_vector</tt> module is done well and reflects all
|
|
C++ functions that are useful at the Python level, for all C++ built-in
|
|
data types (<tt>std_vector.int</tt>, <tt>std_vector.long</tt>, etc.).
|
|
|
|
<p>
|
|
Suppose further that there is statistic module with a C++ class that
|
|
has constructors or member functions that use or return a
|
|
<tt>std::vector</tt>. For example:
|
|
|
|
<pre>
|
|
class xy {
|
|
public:
|
|
xy(const std::vector<double>& x, const std::vector<double>& y) : m_x(x), m_y(y) {}
|
|
const std::vector<double>& x() const { return m_x; }
|
|
const std::vector<double>& y() const { return m_y; }
|
|
double correlation();
|
|
private:
|
|
std::vector<double> m_x;
|
|
std::vector<double> m_y;
|
|
}
|
|
</pre>
|
|
|
|
What is more natural than reusing the <tt>std_vector</tt> extension
|
|
module to expose these constructors or functions to Python?
|
|
|
|
<p>
|
|
Unfortunately, what seems natural needs a little work in both the
|
|
<tt>std_vector</tt> and the <tt>statistics</tt> module.
|
|
|
|
<p>
|
|
In the <tt>std_vector</tt> extension module,
|
|
<tt>std::vector<double></tt> is exposed to Python in the usual
|
|
way with the <tt>class_builder<></tt> template. To also enable the
|
|
automatic conversion of <tt>std::vector<double></tt> function
|
|
arguments or return values in other Boost.Python C++ modules, the
|
|
converters that convert a <tt>std::vector<double></tt> C++ object
|
|
to a Python object and vice versa (i.e. the <tt>to_python()</tt> and
|
|
<tt>from_python()</tt> template functions) have to be exported. For
|
|
example:
|
|
|
|
<pre>
|
|
#include <boost/python/cross_module.hpp>
|
|
//...
|
|
class_builder<std::vector<double> > v_double(std_vector_module, "double");
|
|
export_converters(v_double);
|
|
</pre>
|
|
|
|
In the extension module that wraps <tt>class xy</tt> we can now import
|
|
these converters with the <tt>import_converters<></tt> template.
|
|
For example:
|
|
|
|
<pre>
|
|
#include <boost/python/cross_module.hpp>
|
|
//...
|
|
import_converters<std::vector<double> > v_double_converters("std_vector", "double");
|
|
</pre>
|
|
|
|
That is all. All the attributes that are defined for
|
|
<tt>std_vector.double</tt> in the <tt>std_vector</tt> Boost.Python
|
|
module will be available for the returned objects of <tt>xy.x()</tt>
|
|
and <tt>xy.y()</tt>. Similarly, the constructor for <tt>xy</tt> will
|
|
accept objects that were created by the <tt>std_vector</tt>module.
|
|
|
|
<hr>
|
|
<h2>Placement of <tt>import_converters<></tt> template instantiations</h2>
|
|
|
|
<tt>import_converts<></tt> can be viewed as a drop-in replacement
|
|
for <tt>class_wrapper<></tt>, and the recommendations for the
|
|
placement of <tt>class_wrapper<></tt> template instantiations
|
|
also apply to to <tt>import_converts<></tt>. In particular, it is
|
|
important that an instantiation of <tt>class_wrapper<></tt> is
|
|
visible to any code which wraps a C++ function with a <tt>T</tt>,
|
|
<tt>T*</tt>, const <tt>T&</tt>, etc. parameter or return value.
|
|
Therefore you may want to group all <tt>class_wrapper<></tt> and
|
|
<tt>import_converts<></tt> instantiations at the top of your
|
|
module's init function, then <tt>def()</tt> the member functions later
|
|
to avoid problems with inter-class dependencies.
|
|
|
|
<hr>
|
|
<h2>Non-copyable types</h2>
|
|
|
|
<tt>export_converters()</tt> instantiates C++ template functions that
|
|
invoke the copy constructor of the wrapped type. For a type that is
|
|
non-copyable this will result in compile-time error messages. In such a
|
|
case, <tt>export_converters_noncopyable()</tt> can be used to export
|
|
the converters that do not involve the copy constructor of the wrapped
|
|
type. For example:
|
|
|
|
<pre>
|
|
class_builder<store> py_store(your_module, "store");
|
|
export_converters_noncopyable(py_store);
|
|
</pre>
|
|
|
|
The corresponding <tt>import_converters<></tt> statement does not
|
|
need any special attention:
|
|
|
|
<pre>
|
|
import_converters<store> py_store("noncopyable_export", "store");
|
|
</pre>
|
|
|
|
<hr>
|
|
<h2>Python module search path</h2>
|
|
|
|
The <tt>std_vector</tt> and <tt>statistics</tt> modules can now be used
|
|
in the following way:
|
|
|
|
<pre>
|
|
import std_vector
|
|
import statistics
|
|
x = std_vector.double([1, 2, 3, 4])
|
|
y = std_vector.double([2, 4, 6, 8])
|
|
xy = statistics.xy(x, y)
|
|
xy.correlation()
|
|
</pre>
|
|
|
|
In this example it is clear that Python has to be able to find both the
|
|
<tt>std_vector</tt> and the <tt>statistics</tt> extension module. In
|
|
other words, both extension modules need to be in the Python module
|
|
search path (<tt>sys.path</tt>).
|
|
|
|
<p>
|
|
The situation is not always this obvious. Suppose the
|
|
<tt>statistics</tt> module has a <tt>random()</tt> function that
|
|
returns a vector of random numbers with a given length:
|
|
|
|
<pre>
|
|
import statistics
|
|
x = statistics.random(5)
|
|
y = statistics.random(5)
|
|
xy = statistics.xy(x, y)
|
|
xy.correlation()
|
|
</pre>
|
|
|
|
A naive user will not easily anticipate that the <tt>std_vector</tt>
|
|
module is used to pass the <tt>x</tt> and <tt>y</tt> vectors around. If
|
|
the <tt>std_vector</tt> module is in the Python module search path,
|
|
this form of ignorance is of no harm. On the contrary, we are glad
|
|
that we do not have to bother the user with details like this.
|
|
|
|
<p>
|
|
If the <tt>std_vector</tt> module is not in the Python module search
|
|
path, a Python exception will be raised:
|
|
|
|
<pre>
|
|
Traceback (innermost last):
|
|
File "foo.py", line 2, in ?
|
|
x = statistics.random(5)
|
|
ImportError: No module named std_vector
|
|
</pre>
|
|
|
|
As is the case with any system of a non-trivial complexity, it is
|
|
important that the setup is consistent and complete.
|
|
|
|
<hr>
|
|
<h2>Two-way module dependencies</h2>
|
|
|
|
Boost.Python supports two-way module dependencies. This is best
|
|
illustrated by a simple example.
|
|
|
|
<p>
|
|
Suppose there is a module <tt>ivect</tt> that implements vectors of
|
|
integers, and a similar module <tt>dvect</tt> that implements vectors
|
|
of doubles. We want to be able do convert an integer vector to a double
|
|
vector and vice versa. For example:
|
|
|
|
<pre>
|
|
import ivect
|
|
iv = ivect.ivect((1,2,3,4,5))
|
|
dv = iv.as_dvect()
|
|
</pre>
|
|
|
|
The last expression will implicitly import the <tt>dvect</tt> module in
|
|
order to enable the conversion of the C++ representation of
|
|
<tt>dvect</tt> to a Python object. The analogous is possible for a
|
|
<tt>dvect</tt>:
|
|
|
|
<pre>
|
|
import dvect
|
|
dv = dvect.dvect((1,2,3,4,5))
|
|
iv = dv.as_ivect()
|
|
</pre>
|
|
|
|
Now the <tt>ivect</tt> module is imported implicitly.
|
|
|
|
<p>
|
|
Note that the two-way dependencies are possible because the
|
|
dependencies are resolved only when needed. This is, the initialization
|
|
of the <tt>ivect</tt> module does not rely on the <tt>dvect</tt>
|
|
module, and vice versa. Only if <tt>as_dvect()</tt> or
|
|
<tt>as_ivect()</tt> is actually invoked will the corresponding module
|
|
be implicitly imported. This also means that, for example, the
|
|
<tt>dvect</tt> module does not have to be available at all if
|
|
<tt>as_dvect()</tt> is never used.
|
|
|
|
<hr>
|
|
<h2>Clarification of compile-time and link-time dependencies</h2>
|
|
|
|
Boost.Python's support for resolving cross-module dependencies at
|
|
runtime does not imply that compile-time dependencies are eliminated.
|
|
For example, the statistics extension module in the example above will
|
|
need to <tt>#include <vector></tt>. This is immediately obvious
|
|
from the definition of <tt>class xy</tt>.
|
|
|
|
<p>
|
|
If a library is wrapped that consists of both header files and compiled
|
|
components (e.g. <tt>libdvect.a</tt>, <tt>dvect.lib</tt>, etc.), both
|
|
the Boost.Python extension module with the
|
|
<tt>export_converters()</tt> statement and the module with the
|
|
<tt>import_converters<></tt> statement need to be linked against
|
|
the object library. Ideally one would build a shared library (e.g.
|
|
<tt>libdvect.so</tt>, <tt>dvect.dll</tt>, etc.). However, this
|
|
introduces the issue of having to configure the search path for the
|
|
dynamic loading correctly. For small libraries it is therefore often
|
|
more convenient to ignore the fact that the object files are loaded
|
|
into memory more than once.
|
|
|
|
<hr>
|
|
<h2>Summary of motivation for cross-module support</h2>
|
|
|
|
The main purpose of Boost.Python's cross-module support is to allow for
|
|
a modular system layout. With this support it is straightforward to
|
|
reflect C++ code organization at the Python level. Without the
|
|
cross-module support, a multi-purpose module like <tt>std_vector</tt>
|
|
would be impractical because the entire wrapper code would somehow have
|
|
to be duplicated in all extension modules that use it, making them
|
|
harder to maintain and harder to build.
|
|
|
|
<p>
|
|
Another motivation for the cross-module support is that two extension
|
|
modules that wrap the same class cannot both be imported into Python.
|
|
For example, if there are two modules <tt>A</tt> and <tt>B</tt> that
|
|
both wrap a given <tt>class X</tt>, this will work:
|
|
|
|
<pre>
|
|
import A
|
|
x = A.X()
|
|
</pre>
|
|
|
|
This will also work:
|
|
|
|
<pre>
|
|
import B
|
|
x = B.X()
|
|
</pre>
|
|
|
|
However, this will fail:
|
|
|
|
<pre>
|
|
import A
|
|
import B
|
|
python: /net/cci/rwgk/boost/boost/python/detail/extension_class.hpp:866:
|
|
static void boost::python::detail::class_registry<X>::register_class(boost::python::detail::extension_class_base *):
|
|
Assertion `static_class_object == 0' failed.
|
|
Abort
|
|
</pre>
|
|
|
|
A good solution is to wrap <tt>class X</tt> only once. Depending on the
|
|
situation, this could be done by module <tt>A</tt> or <tt>B</tt>, or an
|
|
additional small extension module that only wraps and exports
|
|
<tt>class X</tt>.
|
|
|
|
<p>
|
|
Finally, there can be important psychological or political reasons for
|
|
using the cross-module support. If a group of classes is lumped
|
|
together with many others in a huge module, the authors will have
|
|
difficulties in being identified with their work. The situation is
|
|
much more transparent if the work is represented by a module with a
|
|
recognizable name. This is not just a question of strong egos, but also
|
|
of getting credit and funding.
|
|
|
|
<hr>
|
|
<h2>Why not use <tt>export_converters()</tt> universally?</h2>
|
|
|
|
There is some overhead associated with the Boost.Python cross-module
|
|
support. Depending on the platform, the size of the code generated by
|
|
<tt>export_converters()</tt> is roughly 10%-20% of that generated
|
|
by <tt>class_builder<></tt>. For a large extension module with
|
|
many wrapped classes, this could mean a significant difference.
|
|
Therefore the general recommendation is to use
|
|
<tt>export_converters()</tt> only for classes that are likely to
|
|
be used as function arguments or return values in other modules.
|
|
|
|
<hr>
|
|
© Copyright Ralf W. Grosse-Kunstleve 2001. Permission to copy,
|
|
use, modify, sell and distribute this document is granted provided this
|
|
copyright notice appears in all copies. This document is provided "as
|
|
is" without express or implied warranty, and with no claim as to its
|
|
suitability for any purpose.
|
|
|
|
<p>
|
|
Updated: April 2001
|
|
|
|
</div>
|