Self-referencing class: concrete python class from C interface

Question

I am trying to design a C interface which could easily be extended in Python (using ctypes). I've used the natural idiom in C:

struct format {
    int (*can_open)(const char *filename);
    struct format * (*open)(const char *filename);
    void (*delete)(struct format *self);
    int (*read)(struct format *self, char *buf, size_t len);
};

It works nicely if I want to extend this interface from C directly:

struct derived /* concrete implementation */
{
    struct format base;
};

But what I would really like to do, is implement this interface from Python using ctypes. Here is what I have so far:

CANOPENFUNC   = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_char_p)
#OPENFUNC     = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_char_p)
#OPENFUNC     = ctypes.CFUNCTYPE(ctypes.POINTER( python_format ), ctypes.c_char_p)
#DELETEFUNC   = ctypes.CFUNCTYPE(None, ctypes.c_void_p)
#READFUNC     = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_void_p)

def py_canopen_func( string ):
    print "py_canopen_func", string
    return 1

canopen_func   = CANOPENFUNC(py_canopen_func)
#open_func     = OPENFUNC(  py_open_func)
#delete_func   = DELETEFUNC(py_canopen_func)
#read_func     = READFUNC(py_canopen_func)

class python_format(ctypes.Structure):
  _fields_ = (
    ('can_open',  CANOPENFUNC),
    ('open',      OPENFUNC),
    ('delete',    DELETEFUNC),
    ('read',      READFUNC),
  )
  def __init__(self):
    self.can_open = canopen_func
    OPENFUNC    = ctypes.CFUNCTYPE(ctypes.POINTER(python_format), ctypes.c_char_p)
    def py_open_func2( string ):
      print "py_open_func2", string
      return ctypes.byref(self)
    self.open   = OPENFUNC( py_open_func2 )
    #self.delete = delete_func
    #self.read = read_func

Really I am struggling to define the prototype for OPENFUNC here. Technically it should be:

OPENFUNC    = ctypes.CFUNCTYPE(ctypes.POINTER(python_format), ctypes.c_char_p)

However I need to define python_format first, which in turns requires a definition for OPENFUNC.

Bonus point: what would be an actual function implementation ? For instance:

def func( str ): return None

or

def func( str ): i = python_format(); return ctypes.pointer(i)

both gives me:

class python_format(ctypes.Structure):
  pass
OPENFUNC = ctypes.CFUNCTYPE(ctypes.POINTER( python_format ), ctypes.c_char_p)
OPENFUNC( func )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: invalid result type for callback function

Is this related to this other issue ? If so should I change my initial C design, since I will not be able to return a pointer to a python_format instance from a callback ?

Correct. `open` is meant to return the allocated instance. Ideally `open` implementation in python, should simple `return self` or `return ctypes.pointer(self)`. `delete` will be a no-op in python because of the garbage collector. — malat, Apr 15 '15 at 08:35
The `TypeError` message you get when trying to use a non-simple type as the result of a callback is less than helpful. A callback's result type has to have a `setfunc` in its `StgDictObject` (a ctypes extension of the regular `PyDictObject`). This requirement restricts you to using a simple type such as `c_void_p`, and the callback function should return `ctypes.addressof(self)`. — Eryk Sun, Apr 15 '15 at 09:57

martineau · Answer 1 · 2020-03-03T21:22:03.237

In the documentation for ctypes.Structure._fields_ it explains how to do this:

It is possible to define the _fields_ class variable after the class statement that defines the Structure subclass, this allows to create data types that directly or indirectly reference themselves

This means you can add a:

class python_format(ctypes.Structure):  # forward declaration
    pass

and then after defining OPENFUNC (and the other function types):

OPENFUNC = ctypes.CFUNCTYPE(ctypes.POINTER( python_format ), ctypes.c_char_p)
DELETEFUNC = etc...

be able to then define python_format._fields_ thusly:

python_format._fields_ = (
    ('can_open',  CANOPENFUNC),
    ('open',      OPENFUNC),
    ('delete',    DELETEFUNC),
    ('read',      READFUNC),
  )

Here's a more complete example based on your code:

import ctypes

class python_format(ctypes.Structure):  # forward declaration
    pass

CANOPENFUNC = ctypes.PYFUNCTYPE(ctypes.c_int, ctypes.c_char_p)
OPENFUNC = ctypes.PYFUNCTYPE(ctypes.c_int,
                                 ctypes.POINTER(python_format),
                                 ctypes.c_char_p)
DELETEFUNC = ctypes.PYFUNCTYPE(None, ctypes.c_void_p)
READFUNC = ctypes.PYFUNCTYPE(ctypes.c_int, ctypes.c_void_p)

def py_canopen_func(self, string):
    print "py_canopen_func", string
    return 1

def py_open_func(self, string):
    print "py_open_func2", string
    # Return types from callbacks cannot be anything other than simple
    # datatypes (c_int, c_float, ..., c_void_p). For other datatypes
    # (STRUCTURE, POINTER, ...), ctypes returns the following error
    # "Invalid result type for callback function"
    # see http://bugs.python.org/issue5710
    return 1  # can't return ctypes.byref(self)

canopen_func = CANOPENFUNC(py_canopen_func)
open_func = OPENFUNC(py_open_func)
#delete_func = DELETEFUNC(py_canopen_func)
#read_func = READFUNC(py_canopen_func)

class python_format(ctypes.Structure):
    python_format._fields_ = (
        ('can_open', CANOPENFUNC),
        ('open', OPENFUNC),
        ('delete', DELETEFUNC),
        ('read', READFUNC),
      )

    def __init__(self):
        self.can_open = canopen_func
        self.open = open_func
        #self.delete = delete_func
        #self.read = read_func

pf = python_format()

Could you give an example of such python function to use `OPENFUNC` ? — malat, Apr 15 '15 at 07:59
There's no reason for any of these callbacks to hold the GIL. Before calling back into Python ctypes reacquires the GIL. You're free to use, and really should use, `CFUNCTYPE` here. Also, per my comment above, the `OPENFUNC` result type can be set to `c_void_p`, for which the callback will return `ctypes.addressof(self)`. — Eryk Sun, Apr 15 '15 at 18:07

malat · Accepted Answer · 2015-04-23T12:43:06.880

In order to have a canonical answer, I'll answer my own question, thanks to @eryksun guidance.

So first thing first, while this is not clear from the documentation, one cannot return complex type from a callback function. Therefore, one cannot map a C function pointer:

struct format {
    struct format * (*open)(const char *filename);
};

to

class python_format:
  pass
OPENFUNC    = ctypes.CFUNCTYPE(ctypes.POINTER(python_format), ctypes.c_char_p)
def py_canopen_func( string ):
    return None
open_func     = OPENFUNC(py_open_func)

The above code will gracefully compiles, but at runtime, one gets:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: invalid result type for callback function

The long answer is that:

The TypeError message you get when trying to use a non-simple type as the result of a callback is less than helpful. A callback's result type has to have a setfunc in its StgDictObject (a ctypes extension of the regular PyDictObject). This requirement restricts you to using a simple type such as c_void_p[...]

Therefore the only solution here, as of today, until issue 5710 is fixed is the following:

class python_format(ctypes.Structure):
  __self_ref = []
  def __init__(self):
    self.open      = self.get_open_func()
  # technically should be a @classmethod but since we are self-referencing
  # ourself, this is just a normal method:
  def get_open_func(self):
    def py_open_func( string ):
      python_format.__self_ref.append( self )
      return ctypes.addressof(self)
    return OPENFUNC( py_open_func )
OPENFUNC     = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_char_p)

# delay init required because `read_info` requires a forward declaration:
python_format._fields_ = (
    ('open',      OPENFUNC),
  )

assigning to `_fields_` doesn't work after it has been used. And it seems that the rules have changed w.r.t. that — Antti Haapala, Aug 28 '17 at 05:05

Self-referencing class: concrete python class from C interface

2 Answers2