Special Method and Operator Support

Overview

Py_cpp is able to wrap suitable C++ functions and C++ operators into Python operators. It supports all of the standard special method names supported by real Python class instances except __complex__ (more on the reasons below). Supported operators include general, numeric, and sequence and mapping operators. In addition, py_cpp provides a simple way to export member variables and define attributes by means of getters and setters.

General Operators

Python provides a number of special operatos for basic customization of a class:

__repr__:: create a string representation from which the object can be reconstructed
__str__:: create a string representation which is suitable for printing
__cmp__:: three-way compare function, used to implement comparison operators (< etc.)
__hash__:: needed to use the object as a dictionary key (only allowed if __cmp__ is also defined)
__nonzero__:: called if the object is used as a truth value (e.g. in an if statement)
__call__:: make instances of the class callable like a function

If we have a suitable C++ function that supports any of these features, we can export it like any other function, using its Python special name. For example, suppose that class Foo provides a string conversion function:

    std::string to_string(Foo const & f)
    {
        std::ostringstream s;
        s << f;
        return s.str();
    }

This function would be wrapped like this:

    python::class_builder<:Foo> foo_class(my_module, "Foo");
    foo_class.def(&to_string, "__str__");

Note that py_cpp also supports automatic wrapping in case of __str__ and __cmp__. This is explained in the next section and the table of numeric operators.

Numeric Operators

There are two fundamental ways to define numeric operators within py_cpp: manual wrapping (as is done with general operators) and automatic wrapping. Lets start with the second possibility. Suppose, C++ defines a class Int (which might represent an infinite-precision integer) which support addition, so that we can write (in C++):

    Int a, b, c;
    ...
    c = a + b;

To enable the same functionality in Python, we first wrap the Int class as usual:

    python::class_builder<Int> int_class(my_module, "Int");
    int_class.def(python::constructor<>());
    ...

Then we export the addition operator like this:

    int_class.def(python::operators<python::op_add>());

Since Int also supports subtraction, multiplication, adn division, we want to export those also. This can be done in a single command by 'or'ing the operator identifiers together (a complete list of these identifiers and the corresponding operators can be found in the table):

    int_class.def(python::operators<(python::op_sub | python::op_mul | python::op_div)>());

Note that the or-expression must be enclosed in parentheses. This form of operator definition will wrap homogeneous operators, i.e. operators whose left and right operand have the same type. Now, suppose that our C++ library also supports addition of Ints and plain integers:

    Int a, b;
    int i;
    ...
    a = b + i;
    a = i + b;

To wrap these heterogeneous operators (left and right hand side have different types), we need a possibility to specify a different type for one of the operands. This is done using the right_operand and left_operand templates:

    int_class.def(python::operators<python::op_add>(), python::right_operand<int>());
    int_class.def(python::operators<python::op_add>(), python::left_operand<int>());

Py_cpp uses overloading to register several variants of the same operation (more on this in the context of coercion). Again, several operators can be exported at once:

    int_class.def(python::operators<(python::op_sub | python::op_mul | python::op_div)>(),
                       python::right_operand<int>());
    int_class.def(python::operators<(python::op_sub | python::op_mul | python::op_div)>(), 
                       python::left_operand<int>());

The type of the operand not mentioned is taken from the class object. In our example, the class object is int_class, and thus the other operand's type is `Int const &'. You can override this default by explicitly specifying a type in the operators template:

    int_class.def(python::operators<python::op_add, Int>(), python::right_operand<int>());

Here, `Int' would be used instead of `Int const &'.

Note that automatic wrapping doesn't need any specific form of operator+() (or one of the other operators), but rather wraps the expression `left + right'. That is, this mechanism can be used for any definition of operator+(), such as a free function `Int operator+(Int, Int)' or a member function `Int Int::operator+(Int)'.

For the Python operators pow() and abs(), there is no corresponding C++ operator. Instead, automatic wrapping attempts to wrap C++ functions of the same name. This only works if those functions are known in namespace python::detail. Thus it might be necessary to add a using declaration prior to wrapping:

    namespace python { 
      namespace detail {
        using my_namespace::pow;
        using my_namespace::abs;
    }}

In some cases, automatic wrapping of operators is not possible or not desirable. Suppose, for example, that the modulo operation for Ints is defined by a set of functions mod() (for automatic wrapping, we would need operator%()):

    Int mod(Int const & left, Int const & right);
    Int mod(Int const & left, int right);
    Int mod(int left, Int const & right);

In order to create the Python operator "__mod__" from these functions, we have to wrap them manually:

    int_class.def((Int (*)(Int const &, Int const &))&mod, "__mod__");
    int_class.def((Int (*)(Int const &, int))&mod, "__mod__");

The third form (with int as left operand) cannot be wrapped this way. We must first create a function rmod() with the operands reversed:

    Int rmod(Int const & right, int left)
    {
        return mod(left, right);
    }

This function must be wrapped under the name "__rmod__":

    int_class.def(&rmod,  "__rmod__");

A list of the possible operator names is also found in the table. Special treatment is necessary to export the ternary pow operator.

Automatic and manual wrapping can be mixed arbitrarily. Note that you cannot overload the same operator for a given extension class on both `int' and `float', because Python implicitly converts these types into each other. Thus, the overloaded variant found first (be it `int' or `float') will be used for either of the two types.

Coercion

Plain Python can only execute operators with identical types on the left and right hand side. If it encounters an expression where the types of the left and right operand differ, it tries to coerce these type to a common type before invoking the actual operator. Implementing good coercion functions can be difficult if many type combinations must be supported.

In contrast, py_cpp provides overloading. By means of overloading, operator calling can be simplyfied drastically: you just register operators for all desired type combinations, and py_cpp automatically ensures that the correct function is called in each case. User defined coercion functions are not necessary. To enable operator overloading, py_cpp provides a standard coercion which is implicitly registered whenever automatic operator wrapping is used.

If you wrap all operator functions manually, but still want to use operator overloading, you have to register the standard coercion function explicitly:

    // this is not necessary if automatic operator wrapping is used
    int_class.def_standard_coerce();

In case you encounter a situation where you absolutely need a customized coercion, you can overload the "__coerce__" operator itself. The signature of a coercion function must look like this:

    python::tuple custom_coerce(PyObject * left, PyObject * right);

The resulting tuple must contain two elements which represent the values of left and right converted to the same type. Such a function is wrapped as usual:

    some_class.def(&custom_coerce, "__coerce__");

Note that the custom coercion function is only used if it is defined before any automatic operator wrapping on the given class or a call to `some_class.def_standard_coerce()'.

The Ternary `pow()` Operator

In addition to the usual binary pow()-operator (meaning x^y), Python also provides a ternary variant that implements (x^y) % z (presumably using a more efficient algorithm than concatenation of power and modulo operators). Automatic operator wrapping can only be used with the binary variant. Ternary pow() must always be wrapped manually. For a homgeneous ternary pow(), this is done as usual:

    Int power(Int const & first, Int const & second, Int const & module);
    typedef Int (ternary_function1)(const Int&, const Int&, const Int&);
    ...
    int_class.def((ternary_function1)&power,  "__pow__");

In case you want to support this function with non-uniform argument types, wrapping is a little more involved. Suppose, you have to wrap:

    Int power(Int const & first, int second, int module);
    Int power(int first, Int const & second, int module);
    Int power(int first, int second, Int const & module);

The first variant can be wrapped as usual:

    typedef Int (ternary_function2)(const Int&, int, int);
    int_class.def((ternary_function2)&power,  "__pow__");

In the second variant, however, Int appears only as second argument, and in the last one it is the third argument. Therefor we must first provide functions where the argumant order is changed so that Int appears in first place:

    Int rpower(Int const & second, int first, int module)
    {
        return power(first, second, third);
    }
    Int rrpower(Int const & third, int first, int second)
    {
        return power(first, second, third);
    }

These functions must be wrapped under the names "__rpow__" and "__rrpow__" respectively:

    int_class.def((ternary_function2)&rpower,  "__rpow__");
    int_class.def((ternary_function2)&rrpower,  "__rrpow__");

Note that "__rrpow__" is an extension not present in plain Python.

Table of Numeric Operators

Py_cpp supports the Python operators listed in the following table. Note that comparison (__cmp__) and string conversion (__str__) operators are included in the list, although they are not strictly "numeric".

Python Operator Name	Python Expression	C++ Operator Id	C++ Expression Used For Automatic Wrapping with `cpp_left = from_python(left, type<Left>())`, `cpp_right = from_python(right, type<Right>())`, and `cpp_oper = from_python(oper, type<Oper>())`
`__add__, __radd__`	`left + right`	`python::op_add`	`cpp_left + cpp_right`
`__sub__, __rsub__`	`left - right`	`python::op_sub`	`cpp_left - cpp_right`
`__mul__, __rmul__`	`left * right`	`python::op_mul`	`cpp_left * cpp_right`
`__div__, __rdiv__`	`left / right`	`python::op_div`	`cpp_left / cpp_right`
`__mod__, __rmod__`	`left % right`	`python::op_mod`	`cpp_left % cpp_right`
`__divmod__, __rdivmod__`	`(quotient, remainder) = divmod(left, right)`	`python::op_divmod`	`cpp_left / cpp_right` and `cpp_left % cpp_right`
`__pow__, __rpow__`	`pow(left, right)` (binary power)	`python::op_pow`	`pow(cpp_left, cpp_right)`
`__pow__`	`pow(left, right, modulo)` (ternary power modulo)	no automatic wrapping, special treatment required
`__lshift__, __rlshift__`	`left << right`	`python::op_lshift`	`cpp_left << cpp_right`
`__rshift__, __rrshift__`	`left >> right`	`python::op_rshift`	`cpp_left >> cpp_right`
`__and__, __rand__`	`left & right`	`python::op_and`	`cpp_left & cpp_right`
`__xor__, __rxor__`	`left ^ right`	`python::op_xor`	`cpp_left ^ cpp_right`
`__or__, __ror__`	`left \| right`	`python::op_or`	`cpp_left \| cpp_right`
`__cmp__, __rcmp__`	`cmp(left, right)` (3-way compare) `left < right` `left <= right` `left > right` `left >= right` `left == right`	`python::op_cmp`	`cpp_left < cpp_right` and `cpp_right < cpp_left`
`__neg__`	`-oper` (unary negation)	`python::op_neg`	`-cpp_oper`
`__pos__`	`+oper` (identity)	`python::op_pos`	`+cpp_oper`
`__abs__`	`abs(oper)` (absolute value)	`python::op_abs`	`abs(cpp_oper)`
`__invert__`	`~oper` (bitwise inversion)	`python::op_invert`	`~cpp_oper`
`__int__`	`int(oper)` (integer conversion)	`python::op_int`	`long(cpp_oper)`
`__long__`	`long(oper)` (infinite precision integer conversion)	`python::op_long`	`PyLong_FromLong(cpp_oper)`
`__float__`	`float(oper)` (float conversion)	`python::op_float`	`double(cpp_oper)`
`__oct__`	`oct(oper)` (octal conversion)	must be wrapped manually (wrapped function should return a string)
`__hex__`	`hex(oper)` (hex conversion)	must be wrapped manually (wrapped function should return a string)
`__str__`	`str(oper)` (string conversion)	`python::op_str`	`std::ostringstream s; s << oper;`
`__coerce__`	`coerce(left, right)`	usually defined automatically, otherwise special treatment required

Sequence and Mapping Operators

Sequence and mapping operators let wrapped objects behave in accordance to Python's iteration and access protocols. These protocols differ considerably from the ones found in C++. For example, Python's typically iteration idiom looks like "for i in S:" , while in C++ one uses "for(iterator i = S.begin(); i != S.end(); ++i)". One could try to wrap C++ iterators in order to carry the C++ idiom into Python. However, this does not work very well because (1) it leads to non-uniform Python code (wrapped types must be used in a different way than Python built-in types) and (2) iterators are often implemented as plain C++ pointers which cannot be wrapped easily because py_cpp is designed to handle objects only.

Thus, it is a good idea to provide sequence and mapping operators for your wrapped containers. These operators have to be wrapped manually because there are no corresponding C++ operators that could be used for automatic wrapping. The Python documentation lists the relevant container operators. In particular, expose __getitem__, __setitem__ and remember to throw the PyExc_IndexError when the index is out-of-range in order to enable the "for i in S:" idiom.

Here is an example. Suppose you, we want to wrap a std::map<std::size_t,std::string>. This is done as follows as follows:

typedef std::map<std::size_t, std::string> StringMap;

// A helper function for dealing with errors. Throw a Python exception
// if p == m.end().
void throw_key_error_if_end(
        const StringMap& m, 
        StringMap::const_iterator p, 
        std::size_t key)
{
    if (p == m.end())
    {
        PyErr_SetObject(PyExc_KeyError, python::converters::to_python(key));
        throw python::error_already_set();
    }
}

// Define some simple wrapper functions which match the Python  protocol
// for __getitem__, __setitem__, and __delitem__.  Just as in Python, a
// free function with a "self" first parameter makes a fine class method.

const std::string& get_item(const StringMap& self, std::size_t key)
{
    const StringMap::const_iterator p = self.find(key);
    throw_key_error_if_end(self, p, key);
    return p->second;
}

// Sets the item corresponding to key in the map.
void StringMapPythonClass::set_item(StringMap& self, std::size_t key, const std::string& value)
{
    self[key] = value;
}

// Deletes the item corresponding to key from the map.
void StringMapPythonClass::del_item(StringMap& self, std::size_t key)
{
    const StringMap::iterator p = self.find(key);
    throw_key_error_if_end(self, p, key);
    self.erase(p);
}

class_builder<StringMap> string_map(my_module, "StringMap");
string_map.def(python::constructor<>());
string_map.def(&StringMap::size, "__len__");
string_map.def(get_item, "__getitem__");
string_map.def(set_item, "__setitem__");
string_map.def(del_item, "__delitem__");

Then in Python:

>>> m = StringMap()
>>> m[1]
Traceback (innermost last):
  File "<stdin>", line 1, in ?
KeyError: 1
>>> m[1] = 'hello'
>>> m[1]
'hello'
>>> del m[1]
>>> m[1]            # prove that it's gone
Traceback (innermost last):
  File "<stdin>", line 1, in ?
KeyError: 1
>>> del m[2]
Traceback (innermost last):
  File "<stdin>", line 1, in ?
KeyError: 2
>>> len(m)
0
>>> m[0] = 'zero'
>>> m[1] = 'one'
>>> m[2] = 'two'
>>> m[3] = 'three'
>>> len(m)
4
>>> for i in m:
...    print i
...
zero
one
two
three

Getters and Setters

Py_cpp extension classes support some additional "special method" protocols not supported by built-in Python classes. Because writing __getattr__, __setattr__, and __delattr__ functions can be tedious in the common case where the attributes being accessed are known statically, py_cpp checks the special names

__getattr__<name>__
__setattr__<name>__
__delattr__<name>__

to provide functional access to the attribute <name>. This facility can be used from C++ or entirely from Python. For example, the following shows how we can implement a "computed attribute" in Python:

>>> class Range(AnyPy_cppExtensionClass):
...    def __init__(self, start, end):
...        self.start = start
...        self.end = end
...    def __getattr__length__(self):
...        return self.end - self.start
...
>>> x = Range(3, 9)
>>> x.length
6

Direct Access to Data Members

Py_cpp uses the special __xxxattr__<name>__ functionality described above to allow direct access to data members through the following special functions on class_builder<> and extension_class<>:

def_getter(pointer-to-member, name) // read access to the member via attribute name
def_setter(pointer-to-member, name) // write access to the member via attribute name
def_readonly(pointer-to-member, name) // read-only access to the member via attribute name
def_read_write(pointer-to-member, name) // read/write access to the member via attribute name

Note that the first two functions, used alone, may produce surprising behavior. For example, when def_getter() is used, the default functionality for setattr() and delattr() remains in effect, operating on items in the extension obj's name-space (i.e., its __dict__). For that reason, you'll usually want to stick with def_readonly and def_read_write.

For example, to expose a std::pair<int,long> we might write:

typedef std::pair<int,long> Pil;
int first(const Pil& x) { return x.first; }
long second(const Pil& x) { return x.second; }
   ...
my_module.def(first, "first");
my_module.def(second, "second");

class_builder<Pil> pair_int_long(my_module, "Pair");
pair_int_long.def(python::constructor<>());
pair_int_long.def(python::constructor<int,long>());
pair_int_long.def_read_write(&Pil::first, "first");
pair_int_long.def_read_write(&Pil::second, "second");

Now your Python class has attributes first and second which, when accessed, actually modify or reflect the values of corresponding data members of the underlying C++ object. Now in Python:

>>> x = Pair(3,5)
>>> x.first
3
>>> x.second
5
>>> x.second = 8
>>> x.second
8
>>> second(x) # Prove that we're not just changing the obj __dict__
8

And what about `complex`?

That, dear reader, is one problem we don't know how to solve. The Python source contains the following fragment, indicating the special-case code really is hardwired:

/* XXX Hack to support classes with __complex__ method */
if (PyInstance_Check(r)) { ...

Previous: Inheritance Next: A Peek Under the Hood Up: Top

© Copyright David Abrahams and Ullrich K�the 2000. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.

Updated: Nov 21, 2000