|
|
- |
+ |
By now you should know how to use Boost.Python to call your C++ code from Python. However sometimes you might need to do the reverse: call Python code from the C++-side. This requires you to 'embed' the Python interpreter in your C++ program. For this we need to use the Python C API.
-The basics to embedding with the API are pretty simple. First you need to make sure that your program links with pythonXY.lib where X.Y is your Python version number. You'll typically find this library in the libs subdirectory of your Python installation. Now your program can follow these steps:
-Of course, there can be other C++ code between all of these steps.
--Now this looks pretty simple, but we'll see in the next section that step 3 can be tricky.
--Most things in Python are objects. Therefore it is only natural that many of the Python C API functions operate on Python objects. Because C/C++ can't work with Python objects directly, the API defines a PyObject structure and a lot of functions to operate on PyObject pointers.
--An important property of Python objects, and therefore of PyObjects, is that they are reference counted. This has major advantages compared to 'dumb' copying: it requiring less memory and it avoids unnecessary copying overhead. However, there is a downside as well. Although the reference counting is transparent from the Python-side, it is quite explicit in the C API. In other words you must increase and decrease the reference counts of PyObjects manually using the -Py_XINCREF and -Py_XDECREF macros. This is cumbersome, and if you don't do it properly some objects might be released when you still need them, or not be released at all.
--I will briefly explain how to update the reference counts correctly, but I'll soon show a better way to do things.
--The API functions that return PyObject pointers are listed in the Python C API documentation as either returning a borrowed or a new reference. The difference is in reference ownership.
--When a new reference is returned, you own that reference. Therefore you don't need to worry about the object being deallocated while you still need it. You do need to decrease the reference count when you are done with it however, otherwise the object will never be deallocated. In other words, you'll have a resource leak.
--Here's a simple example:
-TODO: need different examples because the current ones can be done very naturally with python::tuple which makes them rather unsuitable for explaining python:: -handle
- // Create a new tuple of 3 elements long
- PyObject* my_tuple = PyTuple_New(3);
- ... // Use my_tuple here
- // We're done with the tuple
- Py_XDECREF(my_tuple);
-
-When a borrowed reference is returned, you do not have ownership of the reference. So if you just want to discard the return value, there is nothing you have to do: you didn't own it anyway. If want to use it however, you'll first have to increase its reference count (to prevent the objects deletion). Then later on when you are done with itm you'll need to decrease the reference count again. Here's another example:
-
- // Retrieve the first item in the tuple
- PyObject* first = PyTuple_GetItem(my_tuple, 0);
- Py_XINCREF(first);
- ... // Use first here
- // We're done with the first tuple item
- Py_XDECREF(first);
-
-While this certainly works, it's hardly elegant and it's easy to make mistakes, especially when there are multiple execution paths.
--Now we get to the good news. If you don't want to do all the error prone reference counting yourself, you can let Boost.Python do all the work. First include <boost/python.hpp> instead of "Python.h" and link to boost_python.lib (or boost_python_debug.lib) instead of pythonXY.lib. Then all we really need to do is replace every PyObject* with -handle<PyObject> and we can remove all the -Py_XINCREFs and -Py_XDECREFs! All the reference counting will be done automagically through the power of the -Resource Acquisition Is Initialization idiom.
--We still need a way to differentiate between new and borrowed references though. Luckily, this is pretty straightforward using the -borrowed function. Here is an example using -handle that combines the functionality of the above two PyObject* examples. Notice how it is both shorter and cleaner:
-
- // Create a new tuple of 3 elements long
- handle<PyObject> my_tuple( PyTuple_New(3) );
- // Retrieve the first item in the tuple
- handle<PyObject> first( borrowed(PyTuple_GetItem(my_tuple.get(), 0)) );
- ... // Use first here
-
-Note that the -handle member function get() returns the raw PyObject* that is held by the -handle.
-TODO: Explain PyRun_... basics, somewhere. Perhaps these functions can be used in the examples?
-Now that we know how to call Python code from C++ and C++ code from Python, how about doing it both at the same time? Sometimes you might want to call Python code from C++ and have that Python code call C++ code again. If you built your Boost.Python module then you can just use it in your embedded Python code as you would in a standalone Python program: no further changes necessary.
-
-However, you can also define the Boost.Python module in the same program that embeds the Python code which will be using the module. Then you won't have to build the module and place it in the proper directory, and this also prevents others from using it in their own Python code. (Unless they start taking your executable apart that is. 
)
-Doing this is relatively straightforward. You just define your Boost.Python module as usual and use the basic embedding steps described above. However, before calling -Py_Initialize you call -PyImport_AppendInittab first. This function takes the name and initialization function of your module as parameters and adds the module to the interpreters list of builtin modules. So when the Python interpreter comes across an import statement, it will find the module in its list of builtin modules instead of (unsuccessfully) searching for it in the Python directory.
--Your program will look something like this:
-
- BOOST_PYTHON_MODULE(my_module)
- {
- ...
- }
- ...
- PyImport_AppendInittab("my_module", initmy_module);
- Py_Initialize();
- ...
-
-There is one catch at the moment though. You must not call -Py_Finalize. Boost.Python keeps some PyObject references alive in global data structures, and when those go out of scope after interpreter finalization, Python crashes. This will be fixed in the future.
--A more elaborate example showing these techniques is located at -libs/python/test/embedding.cpp.
- |
|
- |
+ |
| + | ++ Embedding basics + | + |
+
|
+  |
+ |
+
+Most things in Python are objects. Therefore it is only natural that many of the Python C API functions operate on Python objects. Because C/C++ can't work with Python objects directly, the API defines a PyObject structure and a lot of functions to operate on PyObject pointers.
++An important property of Python objects, and therefore of PyObjects, is that they are reference counted. This has major advantages compared to 'dumb' copying: it requiring less memory and it avoids unnecessary copying overhead. However, there is a downside as well. Although the reference counting is transparent from the Python-side, it is quite explicit in the C API. In other words you must increase and decrease the reference counts of PyObjects manually using the +Py_XINCREF and +Py_XDECREF macros. This is cumbersome, and if you don't do it properly some objects might be released when you still need them, or not be released at all.
++I will briefly explain how to update the reference counts correctly, but I'll soon show a better way to do things.
++The API functions that return PyObject pointers are listed in the Python C API documentation as either returning a borrowed or a new reference. The difference is in reference ownership.
++When a new reference is returned, you own that reference. Therefore you don't need to worry about the object being deallocated while you still need it. You do need to decrease the reference count when you are done with it however, otherwise the object will never be deallocated: you'll have a resource leak.
++Here's a simple example:
+
+ // Create a new tuple of 3 elements long
+ PyObject* my_tuple = PyTuple_New(3);
+ ... // Use my_tuple here
+ // We're done with the tuple
+ Py_XDECREF(my_tuple);
+
+When a borrowed reference is returned, you do not have ownership of the reference. So if you just want to discard the return value, there is nothing you have to do: you didn't own it anyway. If want to use it however, you'll first have to increase its reference count (to prevent the object's deletion). Then later on when you are done with itm you'll need to decrease the reference count again. Here's another example:
+
+ // Retrieve the first item in the tuple
+ PyObject* first = PyTuple_GetItem(my_tuple, 0);
+ Py_XINCREF(first);
+ ... // Use first here
+ // We're done with the first tuple item
+ Py_XDECREF(first);
+
+While this certainly works, it's hardly elegant and it's easy to make mistakes, especially when there are multiple execution paths.
++To run Python code from C++ there is a family of functions in the API starting with the PyRun_ prefix. You can find the list of these fuctions +here. I shall discuss one of them, the others work very similarly.
+
+ PyObject* PyRun_String(char *str, int start, PyObject *globals, PyObject *locals)
+
+ +PyRun_String takes the code to execute as a null-terminated (C-style) string in its str parameter. The function returns a new reference to the Python object which results from executing the code. If a Python exception occurred during execution, the null pointer is returned.
++The start parameter is the start symbol from the Python grammar to use for interpreting the code. The possible values are:
+| +Start symbols | +|||||
| +Py_eval_input | for interpreting isolated expressions | +Py_file_input | for interpreting sequences of statements | +Py_single_input | for interpreting a single statement |
+Lastly, the globals and locals parameters are Python dictionaries containing the globals and locals of the context in which to run the code. For most intents and purposes you can use the namespace dictionary of the running module for both globals and locals.
++So how do we get a module's namespace dictionary? First we need to get a reference to the module. The function to do this is +PyImport_AddModule. Contrary to what you might guess from its name, it returns a borrowed reference to an existing module.
++Then once we have a reference to the module, we can use +PyModule_GetDict to get a borrowed reference to the module's namespace dictionary.
++Since the running module is usually the __main__ module created upon interpreter initialization, the code to execute a Python program from a string becomes:
+
+ // Get the __main__ module namespace
+ PyObject* main_module = PyImport_AddModule("__main__");
+ Py_XINCREF(main_module);
+ PyObject* main_namespace = PyModule_GetDict(main_module);
+ Py_XINCREF(main_namespace);
+ // Run a single Python expression and retrieve the result
+ PyObject* result = PyRun_String("1 + 1",
+ Py_eval_input, main_namespace, main_namespace);
+ ... // Use the result
+ // Cleanup
+ Py_XDECREF(result);
+ Py_XDECREF(main_namespace);
+ Py_XDECREF(main_module);
+
+ Warning+ +Be careful about what Python code you run and where. Running the addition above on your computer can hardly do any harm. But letting users run arbitrary Python code through your program which is running on a webserver can be a security risk. + |
+
Next up: How Boost.Python can simplify embedding...
|
+ |
+ |
+
Copyright © 2002 Dirk Gerrits
+Permission to copy, use, modify, sell and distribute this document
+ is granted provided this copyright notice appears in all copies. This document
+ is provided "as is" without express or implied warranty, and with
+ no claim as to its suitability for any purpose.
| + | ++ Embedding with Boost.Python + | + |
+
|
+ |
+ |
+
+Now we get to the good news. If you don't want to do all the error prone reference counting yourself, you can let Boost.Python do all the work. First include <boost/python.hpp> instead of "Python.h" and link to boost_python.lib (or boost_python_debug.lib) instead of pythonXY.lib. Then all we really need to do is replace every PyObject* with +handle<> and we can remove all the +Py_XINCREFs and +Py_XDECREFs! All the reference counting will be done automagically through the power of the +Resource Acquisition Is Initialization idiom.
++We still need a way to differentiate between new and borrowed references though. Luckily, this is pretty straightforward using the +borrowed function. Let's rewrite the example to run Python code from a string to use +handles:
+
+ // Get the __main__ module namespace
+ handle<> main_module( borrowed(PyImport_AddModule("__main__")) );
+ handle<> main_namespace( borrowed(PyModule_GetDict(main_module)) );
+ // Run a single Python expression and retrieve the result
+ handle<> result( PyRun_String("1 + 1",
+ Py_eval_input, main_namespace.get(), main_namespace.get()) );
+ ... // Use the result
+
+The +get() member function used here returns the raw PyObject* that is held by the +handle.
++Now that we know how to call Python code from C++ and C++ code from Python, how about doing both at the same time? Sometimes you might want to call Python code from C++ and have that Python code call C++ code again.
++If you built your Boost.Python module and put it in the proper directory, you can just use it in your embedded Python code as you would in a standalone Python program. The embedded interpreter will be able to find your module just as the standalone interpreter would.
+
+However, you can also define the Boost.Python module in the same program that embeds the Python interpreter. Then you won't have to build the module seperately and place it in the proper directory. This also prevents others from using your module in their own Python code. (Unless they start taking your executable apart that is. )
+Doing this is relatively straightforward. You just define your Boost.Python module as usual and use the basic embedding steps described earlier. However, before calling +Py_Initialize you call +PyImport_AppendInittab first.
++This function takes the name and +initialization function of your module as parameters and adds the module to the interpreter's list of builtin modules. So when the Python interpreter comes across an import statement, it will find the module in its list of builtin modules instead of (unsuccessfully) searching for it in the Python directory.
++Your program will look something like this:
+
+ BOOST_PYTHON_MODULE(my_module)
+ {
+ ...
+ }
+ ...
+ PyImport_AppendInittab("my_module", initmy_module);
+ Py_Initialize();
+ ...
+
|
+ Warning + +When creating Boost.Python modules in the same program that embeds interpreter, you must not call +Py_Finalize. Boost.Python keeps some PyObject references alive in global data structures, and when those go out of scope after interpreter finalization, Python crashes. This will be fixed in the future. Stay tuned. + |
+
+A more elaborate example showing these techniques is located at +/libs/python/test/embedding.cpp.
+ |
+ |
+ |
+
Copyright © 2002 Dirk Gerrits
+Permission to copy, use, modify, sell and distribute this document
+ is granted provided this copyright notice appears in all copies. This document
+ is provided "as is" without express or implied warranty, and with
+ no claim as to its suitability for any purpose.