mirror of
https://github.com/boostorg/python.git
synced 2026-01-24 06:02:14 +00:00
moved from branch ralf_grosse_kunstleve to trunk (was cross_module_dependencies.html)
[SVN r9822]
This commit is contained in:
Ralf W. Grosse-Kunstleve
336
doc/cross_module.html
Normal file
336
doc/cross_module.html
Normal file
@@ -0,0 +1,336 @@
|
||||
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0//EN"
|
||||
"http://www.w3.org/TR/REC-html40/strict.dtd">
|
||||
|
||||
<title>Cross-extension-module dependencies</title>
|
||||
|
||||
<div>
|
||||
|
||||
<img src="../../../c++boost.gif"
|
||||
alt="c++boost.gif (8819 bytes)"
|
||||
align="center"
|
||||
width="277" height="86">
|
||||
|
||||
<hr>
|
||||
<h1>Cross-extension-module dependencies</h1>
|
||||
|
||||
It is good programming practice to organize large projects as modules
|
||||
that interact with each other via well defined interfaces. With
|
||||
Boost.Python it is possible to reflect this organization at the C++
|
||||
level at the Python level. This is, each logical C++ module can be
|
||||
organized as a separate Python extension module.
|
||||
|
||||
<p>
|
||||
At first sight this might seem natural and straightforward. However, it
|
||||
is a fairly complex problem to establish cross-extension-module
|
||||
dependencies while maintaining the same ease of use Boost.Python
|
||||
provides for classes that are wrapped in the same extension module. To
|
||||
a large extent this complexity can be hidden from the author of a
|
||||
Boost.Python extension module, but not entirely.
|
||||
|
||||
<hr>
|
||||
<h2>The recipe</h2>
|
||||
|
||||
Suppose there is an extension module that exposes certain instances of
|
||||
the C++ <tt>std::vector</tt> template library such that it can be used
|
||||
from Python in the following manner:
|
||||
|
||||
<pre>
|
||||
import std_vector
|
||||
v = std_vector.double([1, 2, 3, 4])
|
||||
v.push_back(5)
|
||||
v.size()
|
||||
</pre>
|
||||
|
||||
Suppose the <tt>std_vector</tt> module is done well and reflects all
|
||||
C++ functions that are useful at the Python level, for all C++ built-in
|
||||
data types (<tt>std_vector.int</tt>, <tt>std_vector.long</tt>, etc.).
|
||||
|
||||
<p>
|
||||
Suppose further that there is statistic module with a C++ class that
|
||||
has constructors or member functions that use or return a
|
||||
<tt>std::vector</tt>. For example:
|
||||
|
||||
<pre>
|
||||
class xy {
|
||||
public:
|
||||
xy(const std::vector<double>& x, const std::vector<double>& y) : m_x(x), m_y(y) {}
|
||||
const std::vector<double>& x() const { return m_x; }
|
||||
const std::vector<double>& y() const { return m_y; }
|
||||
double correlation();
|
||||
private:
|
||||
std::vector<double> m_x;
|
||||
std::vector<double> m_y;
|
||||
}
|
||||
</pre>
|
||||
|
||||
What is more natural than reusing the <tt>std_vector</tt> extension
|
||||
module to expose these constructors or functions to Python?
|
||||
|
||||
<p>
|
||||
Unfortunately, what seems natural needs a little work in both the
|
||||
<tt>std_vector</tt> and the <tt>statistics</tt> module.
|
||||
|
||||
<p>
|
||||
In the <tt>std_vector</tt> extension module,
|
||||
<tt>std::vector<double></tt> is exposed to Python in the usual
|
||||
way with the <tt>class_builder<></tt> template. To also enable the
|
||||
automatic conversion of <tt>std::vector<double></tt> function
|
||||
arguments or return values in other Boost.Python C++ modules, the
|
||||
converters that convert a <tt>std::vector<double></tt> C++ object
|
||||
to a Python object and vice versa (i.e. the <tt>to_python()</tt> and
|
||||
<tt>from_python()</tt> template functions) have to be exported. For
|
||||
example:
|
||||
|
||||
<pre>
|
||||
#include <boost/python/cross_module.hpp>
|
||||
//...
|
||||
class_builder<std::vector<double> > v_double(std_vector_module, "double");
|
||||
export_converters(v_double);
|
||||
</pre>
|
||||
|
||||
In the extension module that wraps <tt>class xy</tt> we can now import
|
||||
these converters with the <tt>import_converters<></tt> template.
|
||||
For example:
|
||||
|
||||
<pre>
|
||||
#include <boost/python/cross_module.hpp>
|
||||
//...
|
||||
import_converters<std::vector<double> > v_double_converters("std_vector", "double");
|
||||
</pre>
|
||||
|
||||
That is all. All the attributes that are defined for
|
||||
<tt>std_vector.double</tt> in the <tt>std_vector</tt> Boost.Python
|
||||
module will be available for the returned objects of <tt>xy.x()</tt>
|
||||
and <tt>xy.y()</tt>. Similarly, the constructor for <tt>xy</tt> will
|
||||
accept objects that were created by the <tt>std_vector</tt>module.
|
||||
|
||||
<hr>
|
||||
<h2>Placement of <tt>import_converters<></tt> template instantiations</h2>
|
||||
|
||||
<tt>import_converts<></tt> can be viewed as a drop-in replacement
|
||||
for <tt>class_wrapper<></tt>, and the recommendations for the
|
||||
placement of <tt>class_wrapper<></tt> template instantiations
|
||||
also apply to to <tt>import_converts<></tt>. In particular, it is
|
||||
important that an instantiation of <tt>class_wrapper<></tt> is
|
||||
visible to any code which wraps a C++ function with a <tt>T</tt>,
|
||||
<tt>T*</tt>, const <tt>T&</tt>, etc. parameter or return value.
|
||||
Therefore you may want to group all <tt>class_wrapper<></tt> and
|
||||
<tt>import_converts<></tt> instantiations at the top of your
|
||||
module's init function, then <tt>def()</tt> the member functions later
|
||||
to avoid problems with inter-class dependencies.
|
||||
|
||||
<hr>
|
||||
<h2>Non-copyable types</h2>
|
||||
|
||||
<tt>export_converters()</tt> instantiates C++ template functions that
|
||||
invoke the copy constructor of the wrapped type. For a type that is
|
||||
non-copyable this will result in compile-time error messages. In such a
|
||||
case, <tt>export_converters_noncopyable()</tt> can be used to export
|
||||
the converters that do not involve the copy constructor of the wrapped
|
||||
type. For example:
|
||||
|
||||
<pre>
|
||||
class_builder<store> py_store(your_module, "store");
|
||||
export_converters_noncopyable(py_store);
|
||||
</pre>
|
||||
|
||||
The corresponding <tt>import_converters<></tt> statement does not
|
||||
need any special attention:
|
||||
|
||||
<pre>
|
||||
import_converters<store> py_store("noncopyable_export", "store");
|
||||
</pre>
|
||||
|
||||
<hr>
|
||||
<h2>Python module search path</h2>
|
||||
|
||||
The <tt>std_vector</tt> and <tt>statistics</tt> modules can now be used
|
||||
in the following way:
|
||||
|
||||
<pre>
|
||||
import std_vector
|
||||
import statistics
|
||||
x = std_vector.double([1, 2, 3, 4])
|
||||
y = std_vector.double([2, 4, 6, 8])
|
||||
xy = statistics.xy(x, y)
|
||||
xy.correlation()
|
||||
</pre>
|
||||
|
||||
In this example it is clear that Python has to be able to find both the
|
||||
<tt>std_vector</tt> and the <tt>statistics</tt> extension module. In
|
||||
other words, both extension modules need to be in the Python module
|
||||
search path (<tt>sys.path</tt>).
|
||||
|
||||
<p>
|
||||
The situation is not always this obvious. Suppose the
|
||||
<tt>statistics</tt> module has a <tt>random()</tt> function that
|
||||
returns a vector of random numbers with a given length:
|
||||
|
||||
<pre>
|
||||
import statistics
|
||||
x = statistics.random(5)
|
||||
y = statistics.random(5)
|
||||
xy = statistics.xy(x, y)
|
||||
xy.correlation()
|
||||
</pre>
|
||||
|
||||
A naive user will not easily anticipate that the <tt>std_vector</tt>
|
||||
module is used to pass the <tt>x</tt> and <tt>y</tt> vectors around. If
|
||||
the <tt>std_vector</tt> module is in the Python module search path,
|
||||
this form of ignorance is of no harm. On the contrary, we are glad
|
||||
that we do not have to bother the user with details like this.
|
||||
|
||||
<p>
|
||||
If the <tt>std_vector</tt> module is not in the Python module search
|
||||
path, a Python exception will be raised:
|
||||
|
||||
<pre>
|
||||
Traceback (innermost last):
|
||||
File "foo.py", line 2, in ?
|
||||
x = statistics.random(5)
|
||||
ImportError: No module named std_vector
|
||||
</pre>
|
||||
|
||||
As is the case with any system of a non-trivial complexity, it is
|
||||
important that the setup is consistent and complete.
|
||||
|
||||
<hr>
|
||||
<h2>Two-way module dependencies</h2>
|
||||
|
||||
Boost.Python supports two-way module dependencies. This is best
|
||||
illustrated by a simple example.
|
||||
|
||||
<p>
|
||||
Suppose there is a module <tt>ivect</tt> that implements vectors of
|
||||
integers, and a similar module <tt>dvect</tt> that implements vectors
|
||||
of doubles. We want to be able do convert an integer vector to a double
|
||||
vector and vice versa. For example:
|
||||
|
||||
<pre>
|
||||
import ivect
|
||||
iv = ivect.ivect((1,2,3,4,5))
|
||||
dv = iv.as_dvect()
|
||||
</pre>
|
||||
|
||||
The last expression will implicitly import the <tt>dvect</tt> module in
|
||||
order to enable the conversion of the C++ representation of
|
||||
<tt>dvect</tt> to a Python object. The analogous is possible for a
|
||||
<tt>dvect</tt>:
|
||||
|
||||
<pre>
|
||||
import dvect
|
||||
dv = dvect.dvect((1,2,3,4,5))
|
||||
iv = dv.as_ivect()
|
||||
</pre>
|
||||
|
||||
Now the <tt>ivect</tt> module is imported implicitly.
|
||||
|
||||
<p>
|
||||
Note that the two-way dependencies are possible because the
|
||||
dependencies are resolved only when needed. This is, the initialization
|
||||
of the <tt>ivect</tt> module does not rely on the <tt>dvect</tt>
|
||||
module, and vice versa. Only if <tt>as_dvect()</tt> or
|
||||
<tt>as_ivect()</tt> is actually invoked will the corresponding module
|
||||
be implicitly imported. This also means that, for example, the
|
||||
<tt>dvect</tt> module does not have to be available at all if
|
||||
<tt>as_dvect()</tt> is never used.
|
||||
|
||||
<hr>
|
||||
<h2>Clarification of compile-time and link-time dependencies</h2>
|
||||
|
||||
Boost.Python's support for resolving cross-module dependencies at
|
||||
runtime does not imply that compile-time dependencies are eliminated.
|
||||
For example, the statistics extension module in the example above will
|
||||
need to <tt>#include <vector></tt>. This is immediately obvious
|
||||
from the definition of <tt>class xy</tt>.
|
||||
|
||||
<p>
|
||||
If a library is wrapped that consists of both header files and compiled
|
||||
components (e.g. <tt>libdvect.a</tt>, <tt>dvect.lib</tt>, etc.), both
|
||||
the Boost.Python extension module with the
|
||||
<tt>export_converters()</tt> statement and the module with the
|
||||
<tt>import_converters<></tt> statement need to be linked against
|
||||
the object library. Ideally one would build a shared library (e.g.
|
||||
<tt>libdvect.so</tt>, <tt>dvect.dll</tt>, etc.). However, this
|
||||
introduces the issue of having to configure the search path for the
|
||||
dynamic loading correctly. For small libraries it is therefore often
|
||||
more convenient to ignore the fact that the object files are loaded
|
||||
into memory more than once.
|
||||
|
||||
<hr>
|
||||
<h2>Summary of motivation for cross-module support</h2>
|
||||
|
||||
The main purpose of Boost.Python's cross-module support is to allow for
|
||||
a modular system layout. With this support it is straightforward to
|
||||
reflect C++ code organization at the Python level. Without the
|
||||
cross-module support, a multi-purpose module like <tt>std_vector</tt>
|
||||
would be impractical because the entire wrapper code would somehow have
|
||||
to be duplicated in all extension modules that use it, making them
|
||||
harder to maintain and harder to build.
|
||||
|
||||
<p>
|
||||
Another motivation for the cross-module support is that two extension
|
||||
modules that wrap the same class cannot both be imported into Python.
|
||||
For example, if there are two modules <tt>A</tt> and <tt>B</tt> that
|
||||
both wrap a given <tt>class X</tt>, this will work:
|
||||
|
||||
<pre>
|
||||
import A
|
||||
x = A.X()
|
||||
</pre>
|
||||
|
||||
This will also work:
|
||||
|
||||
<pre>
|
||||
import B
|
||||
x = B.X()
|
||||
</pre>
|
||||
|
||||
However, this will fail:
|
||||
|
||||
<pre>
|
||||
import A
|
||||
import B
|
||||
python: /net/cci/rwgk/boost/boost/python/detail/extension_class.hpp:866:
|
||||
static void boost::python::detail::class_registry<X>::register_class(boost::python::detail::extension_class_base *):
|
||||
Assertion `static_class_object == 0' failed.
|
||||
Abort
|
||||
</pre>
|
||||
|
||||
A good solution is to wrap <tt>class X</tt> only once. Depending on the
|
||||
situation, this could be done by module <tt>A</tt> or <tt>B</tt>, or an
|
||||
additional small extension module that only wraps and exports
|
||||
<tt>class X</tt>.
|
||||
|
||||
<p>
|
||||
Finally, there can be important psychological or political reasons for
|
||||
using the cross-module support. If a group of classes is lumped
|
||||
together with many others in a huge module, the authors will have
|
||||
difficulties in being identified with their work. The situation is
|
||||
much more transparent if the work is represented by a module with a
|
||||
recognizable name. This is not just a question of strong egos, but also
|
||||
of getting credit and funding.
|
||||
|
||||
<hr>
|
||||
<h2>Why not use <tt>export_converters()</tt> universally?</h2>
|
||||
|
||||
There is some overhead associated with the Boost.Python cross-module
|
||||
support. Depending on the platform, the size of the code generated by
|
||||
<tt>export_converters()</tt> is roughly 10%-20% of that generated
|
||||
by <tt>class_builder<></tt>. For a large extension module with
|
||||
many wrapped classes, this could mean a significant difference.
|
||||
Therefore the general recommendation is to use
|
||||
<tt>export_converters()</tt> only for classes that are likely to
|
||||
be used as function arguments or return values in other modules.
|
||||
|
||||
<hr>
|
||||
© Copyright Ralf W. Grosse-Kunstleve 2001. Permission to copy,
|
||||
use, modify, sell and distribute this document is granted provided this
|
||||
copyright notice appears in all copies. This document is provided "as
|
||||
is" without express or implied warranty, and with no claim as to its
|
||||
suitability for any purpose.
|
||||
|
||||
<p>
|
||||
Updated: April 2001
|
||||
|
||||
</div>
|
||||
Reference in New Issue
Block a user