2

How to compile a whole Python library along with it's dependencies so that it can be used in C (without invoking Python's runtime). That is, the compiled code has the Python interpreter embedded and Python does not need to be installed on the system.

From my understanding, when Python code is compiled using Cython it:

  • Does not invoke the python runtime if the --embed argument is used
  • Compiles files individually
  • Allows for different modules to be called (from the Python runtime / other compiled Cython files)

The question which are still unclear are:

  • How to use these module files from C? Can the compiled Python files call other compiled Python files when used in C?
  • Does only the library entry point need to be declared or do all functions need to be declared?
  • How to manage the Python dependencies? how to compile them too (so that the Python runtime is not needed).

A simplified example for a python library called module where __init__.py is an empty file:

module/
├── run.py
├── http/
│   ├── __init__.py
│   ├── http_request.py

http_requests.py contains:

import requests

def get_ip():
    r = requests.get('https://ipinfo.io/ip')
    print(r.text)

and run.py contains the following:

from http import http_request

if __name__ == '__main__':
    http_request.get_ip()

How to call the function get_ip from C without using the Python runtime (needing to have Python installed when running the application).

The above example is very simple. The actual use case is collecting/processing robotics data in C at a high sampling rate. Whilst C is great for basic data processing there are excellent Python libraries which allow for much more comprehensive analysis. The objective would be to call the Python libraries on data which has been partially processed in C. This would allow us to get a much more detailed understanding of the data (and process it in "real time"). The data frameworks are way too large for our team to rewrite in C.

Greg
  • 6,043
  • 8
  • 41
  • 93
  • 1
    Related: [Compiling python into a shared library](https://stackoverflow.com/questions/15844383/compiling-python-into-a-shared-library) – metatoaster May 04 '18 at 04:49
  • What is your Python library doing? What python features, modules and primitives does it use? Show if possible some [MCVE] in your question. So **edit your question** to improve it – Basile Starynkevitch May 04 '18 at 04:56
  • @BasileStarynkevitch I've updated the post and included a simple example of a Python library which I would like to call from C without using the Python runtime. – Greg May 04 '18 at 05:12
  • 1
    You should explain in your question why exactly you want to avoid the Python runtime and interpreter.. You should also give more context, and explain what is your library doing. Without more explanations your question is **unclear**. Please edit it again. – Basile Starynkevitch May 04 '18 at 05:35
  • 1
    For your use case, it's practically impossible and/or silly. Why are you/your collaborators writing C code to integrate with this? If it's going to be in an embedded system just write all C code and ditch Python. If it's for some higher level usage just write the whole thing in Python. – metatoaster May 04 '18 at 05:37
  • @metatoaster The example was only trying to show the concept, I've updated the post with the actual use case. – Greg May 04 '18 at 05:47
  • Easiest way is to keep the two separate, have the C code connect to some server and dump the output (or even as simple as write to stdout) and have your Python stack run a server to receive the input for processing (or as simple as read from stdin). – metatoaster May 04 '18 at 05:50
  • @BasileStarynkevitch I've included what the code is doing, and that the reason for wanting to avoid the Python runtime is due to a high sampling/processing rate ("real time") robotics data. – Greg May 04 '18 at 05:50
  • That stills does not explain, and does not justify, why you need to avoid Python. You could rewrite the low level layers of your code in C, and keep Python for higher level stuff (and this is very often done), perhaps running in some different process – Basile Starynkevitch May 04 '18 at 05:52
  • Your claim or wish to avoid Python smells a lot like some [XY problem](http://xyproblem.info/). I don't understand why you don't want any Python runtime on your computer – Basile Starynkevitch May 04 '18 at 06:18
  • @BasileStarynkevitch I agree with you and it's a valid observation. The more often data is processed the better we can control motors (e.g. balancing) which results in smoother actions and lower power usage. The objective is to process the data as frequently as possible whilst not rebuilding the libraries from scratch. Connecting Python and C is not something we've attempted before so compiling Python to C and using that as a library seems like an appropriate solution. – Greg May 04 '18 at 06:31
  • I believe *you are wrong* -or at least very naïve- in thinking that "compiling Python to C and using that as a library seems like an appropriate solution" (I think it is a wrong approach). I think you'll better keep Python and continue using it for high-level parts – Basile Starynkevitch May 04 '18 at 11:42

1 Answers1

2

How to compile a whole Python library along with it's dependencies so that it can be used in C (without invoking Python's runtime).

This is impossible in general. Python code is practically expected to run on a Python interpreter.

Sometimes, when only a small subset of Python is used (even indirectly by everything your Python code is using) you might use Cython (which is actually a superset of a small subset of Python: a lot of genuine Python features cannot be used from Cython, or uses the Python interpreter). But not every Python code can be cythonized, since Python and C have a very different (and incompatible) semantics (and memory management).

Otherwise (and most often), the C code using your Python stuff should embed the Python interpreter.

A wiser and more robust approach, if your goal is to make a self-sufficient C library usable from many C programs (on systems without Python), is to rewrite your code in C.

You could also consider starting (in your C library) some Python process (server-like, doing your Python stuff) and using inter-process communication facilities, that would be operating system specific. Of course Python needs to be installed on the system of the application using your library. For example, for Linux, you might fork some Python process in your library, and use pipe(7) or unix(7) sockets to communication from the C library to that process (perhaps using something like JSONRPC).

Your edit (still not an MCVE) shows some HTTP interaction done in Python. You could consider doing that in C, with the help of HTTP client libraries in C like libcurl, or (if so needed) of HTTP server libraries like libonion.

So consider rewriting your stuff in C but using several existing C libraries (how and what to choose is a very different question, probably off-topic on StackOverflow). Otherwise, accept the dependencies on Python.

The actual use case is collecting/processing robotics data in C at a high sampling rate. Whilst C is great for basic data processing there are excellent Python libraries which allow for much more comprehensive analysis.

You could keep high-level things in Python (see this) but recode low level things in C to accelerate them (many software are doing that, e.g. TensorFlow, ...), perhaps as extensions in C for Python or in some other process. Of course, that means some development efforts. I don't think that avoiding Python entirely is reasonable (getting rid of Python entirely is not pragmatical), if you use a lot of code in Python. BTW, you might perhaps consider embedding some other language in your C application (e.g. Lua, Guile, Ocaml - all of them are rumored to be faster than Python) and keep Python for the higher level, running in some other process.

You need to put more efforts on the architectural design of your thing. I'm not sure that avoiding Python entirely is a wise thing to do. Mixing Python and C (perhaps by having several processes cooperating) could be wiser. Of course you'll have operating system specific stuff (notably on the C side, for inter process communication). If on Linux, read something about Linux system programming in C, e.g. ALP or something newer.

Basile Starynkevitch
  • 1
  • 16
  • 251
  • 479
  • `But not every Python code can be cythonized`: I thought that because Cython is a superset of Python all Python code can be Cythonized? – Greg May 04 '18 at 05:15
  • You probably thought naively, but indeed the Cython site is a bit optimistic... Cython is a superset of a subset of Python – Basile Starynkevitch May 04 '18 at 05:17
  • I'd argue with your description of Cython. It isn't perfectly compatible but the vast majority of Python code _should_ work unchanged. It still does this by calling libpython though so it isn't really translating to C. – DavidW May 04 '18 at 06:15
  • @BasileStarynkevitch - Your assertions regarding Cython are incorrect. In fact, it is more compatible with Python code than the python interpreter itself as it supports all python syntax from 2.6 - 3.5 and can generate equivalent C-API code for it. What superset it may or may not support is irrelevant when discussing pure python code. – danny May 04 '18 at 10:38