+
++
Cross-extension-module dependencies
+ +It is good programming practice to organize large projects as modules +that interact with each other via well defined interfaces. With +Boost.Python it is possible to reflect this organization at the C++ +level at the Python level. This is, each logical C++ module can be +organized as a separate Python extension module. + ++At first sight this might seem natural and straightforward. However, it +is a fairly complex problem to establish cross-extension-module +dependencies while maintaining the same ease of use Boost.Python +provides for classes that are wrapped in the same extension module. To +a large extent this complexity can be hidden from the author of a +Boost.Python extension module, but not entirely. + +
+
The recipe
+ +Suppose there is an extension module that exposes certain instances of +the C++ std::vector template library such that it can be used +from Python in the following manner: + ++import std_vector +v = std_vector.double([1, 2, 3, 4]) +v.push_back(5) +v.size() ++ +Suppose the std_vector module is done well and reflects all +C++ functions that are useful at the Python level, for all C++ built-in +data types (std_vector.int, std_vector.long, etc.). + +
+Suppose further that there is statistic module with a C++ class that +has constructors or member functions that use or return a +std::vector. For example: + +
+class xy {
+ public:
+ xy(const std::vector<double>& x, const std::vector<double>& y) : m_x(x), m_y(y) {}
+ const std::vector<double>& x() const { return m_x; }
+ const std::vector<double>& y() const { return m_y; }
+ double correlation();
+ private:
+ std::vector<double> m_x;
+ std::vector<double> m_y;
+}
+
+
+What is more natural than reusing the std_vector extension
+module to expose these constructors or functions to Python?
+
++Unfortunately, what seems natural needs a little work in both the +std_vector and the statistics module. + +
+In the std_vector extension module, +std::vector<double> is exposed to Python in the usual +way with the class_builder<> template. To also enable the +automatic conversion of std::vector<double> function +arguments or return values in other Boost.Python C++ modules, the +converters that convert a std::vector<double> C++ object +to a Python object and vice versa (i.e. the to_python() and +from_python() template functions) have to be exported. For +example: + +
+ #include <boost/python/cross_module.hpp> + //... + class_builder<std::vector<double> > v_double(std_vector_module, "double"); + export_converters(v_double); ++ +In the extension module that wraps class xy we can now import +these converters with the import_converters<> template. +For example: + +
+ #include <boost/python/cross_module.hpp>
+ //...
+ import_converters<std::vector<double> > v_double_converters("std_vector", "double");
+
+
+That is all. All the attributes that are defined for
+std_vector.double in the std_vector Boost.Python
+module will be available for the returned objects of xy.x()
+and xy.y(). Similarly, the constructor for xy will
+accept objects that were created by the std_vectormodule.
+
++
Placement of import_converters<> template instantiations
+ +import_converts<> can be viewed as a drop-in replacement +for class_wrapper<>, and the recommendations for the +placement of class_wrapper<> template instantiations +also apply to to import_converts<>. In particular, it is +important that an instantiation of class_wrapper<> is +visible to any code which wraps a C++ function with a T, +T*, const T&, etc. parameter or return value. +Therefore you may want to group all class_wrapper<> and +import_converts<> instantiations at the top of your +module's init function, then def() the member functions later +to avoid problems with inter-class dependencies. + ++
Non-copyable types
+ +export_converters() instantiates C++ template functions that +invoke the copy constructor of the wrapped type. For a type that is +non-copyable this will result in compile-time error messages. In such a +case, export_converters_noncopyable() can be used to export +the converters that do not involve the copy constructor of the wrapped +type. For example: + ++class_builder<store> py_store(your_module, "store"); +export_converters_noncopyable(py_store); ++ +The corresponding import_converters<> statement does not +need any special attention: + +
+import_converters<store> py_store("noncopyable_export", "store");
+
+
++
Python module search path
+ +The std_vector and statistics modules can now be used +in the following way: + ++import std_vector +import statistics +x = std_vector.double([1, 2, 3, 4]) +y = std_vector.double([2, 4, 6, 8]) +xy = statistics.xy(x, y) +xy.correlation() ++ +In this example it is clear that Python has to be able to find both the +std_vector and the statistics extension module. In +other words, both extension modules need to be in the Python module +search path (sys.path). + +
+The situation is not always this obvious. Suppose the +statistics module has a random() function that +returns a vector of random numbers with a given length: + +
+import statistics +x = statistics.random(5) +y = statistics.random(5) +xy = statistics.xy(x, y) +xy.correlation() ++ +A naive user will not easily anticipate that the std_vector +module is used to pass the x and y vectors around. If +the std_vector module is in the Python module search path, +this form of ignorance is of no harm. On the contrary, we are glad +that we do not have to bother the user with details like this. + +
+If the std_vector module is not in the Python module search +path, a Python exception will be raised: + +
+Traceback (innermost last): + File "foo.py", line 2, in ? + x = statistics.random(5) +ImportError: No module named std_vector ++ +As is the case with any system of a non-trivial complexity, it is +important that the setup is consistent and complete. + +
+
Two-way module dependencies
+ +Boost.Python supports two-way module dependencies. This is best +illustrated by a simple example. + ++Suppose there is a module ivect that implements vectors of +integers, and a similar module dvect that implements vectors +of doubles. We want to be able do convert an integer vector to a double +vector and vice versa. For example: + +
+import ivect +iv = ivect.ivect((1,2,3,4,5)) +dv = iv.as_dvect() ++ +The last expression will implicitly import the dvect module in +order to enable the conversion of the C++ representation of +dvect to a Python object. The analogous is possible for a +dvect: + +
+import dvect +dv = dvect.dvect((1,2,3,4,5)) +iv = dv.as_ivect() ++ +Now the ivect module is imported implicitly. + +
+Note that the two-way dependencies are possible because the +dependencies are resolved only when needed. This is, the initialization +of the ivect module does not rely on the dvect +module, and vice versa. Only if as_dvect() or +as_ivect() is actually invoked will the corresponding module +be implicitly imported. This also means that, for example, the +dvect module does not have to be available at all if +as_dvect() is never used. + +
+
Clarification of compile-time and link-time dependencies
+ +Boost.Python's support for resolving cross-module dependencies at +runtime does not imply that compile-time dependencies are eliminated. +For example, the statistics extension module in the example above will +need to #include <vector>. This is immediately obvious +from the definition of class xy. + ++If a library is wrapped that consists of both header files and compiled +components (e.g. libdvect.a, dvect.lib, etc.), both +the Boost.Python extension module with the +export_converters() statement and the module with the +import_converters<> statement need to be linked against +the object library. Ideally one would build a shared library (e.g. +libdvect.so, dvect.dll, etc.). However, this +introduces the issue of having to configure the search path for the +dynamic loading correctly. For small libraries it is therefore often +more convenient to ignore the fact that the object files are loaded +into memory more than once. + +
+
Summary of motivation for cross-module support
+ +The main purpose of Boost.Python's cross-module support is to allow for +a modular system layout. With this support it is straightforward to +reflect C++ code organization at the Python level. Without the +cross-module support, a multi-purpose module like std_vector +would be impractical because the entire wrapper code would somehow have +to be duplicated in all extension modules that use it, making them +harder to maintain and harder to build. + ++Another motivation for the cross-module support is that two extension +modules that wrap the same class cannot both be imported into Python. +For example, if there are two modules A and B that +both wrap a given class X, this will work: + +
+import A +x = A.X() ++ +This will also work: + +
+import B +x = B.X() ++ +However, this will fail: + +
+import A +import B +python: /net/cci/rwgk/boost/boost/python/detail/extension_class.hpp:866: +static void boost::python::detail::class_registry<X>::register_class(boost::python::detail::extension_class_base *): +Assertion `static_class_object == 0' failed. +Abort ++ +A good solution is to wrap class X only once. Depending on the +situation, this could be done by module A or B, or an +additional small extension module that only wraps and exports +class X. + +
+Finally, there can be important psychological or political reasons for +using the cross-module support. If a group of classes is lumped +together with many others in a huge module, the authors will have +difficulties in being identified with their work. The situation is +much more transparent if the work is represented by a module with a +recognizable name. This is not just a question of strong egos, but also +of getting credit and funding. + +
+
Why not use export_converters() universally?
+ +There is some overhead associated with the Boost.Python cross-module +support. Depending on the platform, the size of the code generated by +export_converters() is roughly 10%-20% of that generated +by class_builder<>. For a large extension module with +many wrapped classes, this could mean a significant difference. +Therefore the general recommendation is to use +export_converters() only for classes that are likely to +be used as function arguments or return values in other modules. + ++© Copyright Ralf W. Grosse-Kunstleve 2001. Permission to copy, +use, modify, sell and distribute this document is granted provided this +copyright notice appears in all copies. This document is provided "as +is" without express or implied warranty, and with no claim as to its +suitability for any purpose. + +
+Updated: April 2001 + +