172

is it possible to convert a Python program to C/C++?

I need to implement a couple of algorithms, and I'm not sure if the performance gap is big enough to justify all the pain I'd go through when doing it in C/C++ (which I'm not good at). I thought about writing one simple algorithm and benchmark it against such a converted solution. If that alone is significantly faster than the Python version, then I'll have no other choice than doing it in C/C++.

Aaron Hall
  • 291,450
  • 75
  • 369
  • 312
CrazyFlyingCloseline
  • 1,731
  • 2
  • 11
  • 4
  • 38
    As much as Python loses on benchmarks, keep in mind that that 50x or 100x slowdown is still negible if the calculation finishes in a few seconds in Python, and not even true when you do a lot of I/O or have a horrible algorithm. Rather than asking "how much slower is Python?" you should ask "is Python fast enough?" (and it most propably is, honestly) - that's also faster than benchmarking or asking here. –  Jan 10 '11 at 19:03
  • 1
    Implementing an algorithm in python is quite fast and straight forward...you simply have to do it and then check if it is fast enough. Most times you can optimize **the algorithm** to run much faster using different data structures(dict/sets instead of lists...) or different operations. Anyway optimization should occur **after** you have already implemented a first draft of the algorithm and benchmarked/profiled it. – Bakuriu Jan 10 '11 at 19:09
  • @delnan: in my case it's all about computation time. If the C variant needs x hours less, then I'd invest that time in letting the algorithms run longer/again. I simply want to find out (roughly) how much slower Python would be - if it's just a couple of hours I certainly wouldn't use a language I'm not comfortable with (you can ruin the best solutions to problems with bad implementations :P). – CrazyFlyingCloseline Jan 10 '11 at 19:20
  • @delnan's right about Python probably being fast enough for many things. Even when it slower, the ease of devleopment, maintenance, and future enhancement are important factors to consider. – martineau Jan 10 '11 at 19:32
  • "x hours"? How big is this? Have you benchmarked an implementation? Do you have measurements? Have you profiled the implementation? Or are you trying to prematurely optimize the solution? – S.Lott Jan 10 '11 at 19:59
  • @user395760 if you don't care about cloud costs, sureee – Martin F Nov 02 '20 at 13:54

8 Answers8

149

If the C variant needs x hours less, then I'd invest that time in letting the algorithms run longer/again

"invest" isn't the right word here.

  1. Build a working implementation in Python. You'll finish this long before you'd finish a C version.

  2. Measure performance with the Python profiler. Fix any problems you find. Change data structures and algorithms as necessary to really do this properly. You'll finish this long before you finish the first version in C.

  3. If it's still too slow, manually translate the well-designed and carefully constructed Python into C.

    Because of the way hindsight works, doing the second version from existing Python (with existing unit tests, and with existing profiling data) will still be faster than trying to do the C code from scratch.

This quote is important.

Thompson's Rule for First-Time Telescope Makers
It is faster to make a four-inch mirror and then a six-inch mirror than to make a six-inch mirror.

Bill McKeenan
Wang Institute

pradyunsg
  • 13,385
  • 10
  • 36
  • 80
S.Lott
  • 359,791
  • 75
  • 487
  • 757
129

Yes. Look at Cython. It does just that: Converts Python to C for speedups.

Lennart Regebro
  • 147,792
  • 40
  • 207
  • 241
  • 6
    Of course that won't save you anything unless you add a bunch of `cdef` declarations and thereby introduce static typing (otherwise you just juggle opaque `PyObject *` stuff). And it will never get quite as fast as plain C because it's usually interfacing with Python (100% or more? only for plain numerical code that doesn't interface with Python at all for the most time!). But other than that, yes, it can get you a pretty devent speedup. –  Jan 10 '11 at 19:00
  • 8
    @delnan: In fact, it does save you something. Most pure Python code will be faster after compiled. But yes, with the cdefs and static typing you really start seeing differences. And the interfacing with Python you get in all cases where you use C from Python. – Lennart Regebro Jan 10 '11 at 19:30
32

Shed Skin is "a (restricted) Python-to-C++ compiler".

ephemient
  • 180,829
  • 34
  • 259
  • 378
  • 3
    +1 one advantage of Shed Skin is *type inference*: if it is possible to go guess variable types from the program flow, dynamic type-checking is avoided. This typically leads to shorter C++ code that it is actually possible to read and compiles to faster programs. – Kyss Tao Mar 30 '12 at 14:19
  • 1
    There is also [Python → 11l → C++ transpiler](http://11l-lang.org), which is also a restricted Python to C++ compiler, but it supports some Python features, which is not supported with Shed Skin (e.g. nested functions/closures). – tav Dec 11 '18 at 10:27
20

Just came across this new tool in hacker news.

From their page - "Nuitka is a good replacement for the Python interpreter and compiles every construct that CPython 2.6, 2.7, 3.2 and 3.3 offer. It translates the Python into a C++ program that then uses "libpython" to execute in the same way as CPython does, in a very compatible way."

Jordan.J.D
  • 7,382
  • 9
  • 44
  • 75
seagull1089
  • 311
  • 2
  • 4
  • This project is so much more mature than other similar options. It's funny it creates the binary with an `.exe` extension on OSX even though it is a perfectly normal OSX Mach-O executable. Looks like it might be a good replacement for `pyinstaller`, `py2exe`, `py2app`, etc. The `--recurse-***` flags are important to set properly though. – ccpizza Jul 24 '16 at 08:23
  • Nuitka is great, but the C/C++ code created is using PyObject which bind to the CPython-C-code-implementation. It does not produce idiomatic C-code. – Make42 Mar 15 '20 at 11:33
9

I know this is an older thread but I wanted to give what I think to be helpful information.

I personally use PyPy which is really easy to install using pip. I interchangeably use Python/PyPy interpreter, you don't need to change your code at all and I've found it to be roughly 40x faster than the standard python interpreter (Either Python 2x or 3x). I use pyCharm Community Edition to manage my code and I love it.

I like writing code in python as I think it lets you focus more on the task than the language, which is a huge plus for me. And if you need it to be even faster, you can always compile to a binary for Windows, Linux, or Mac (not straight forward but possible with other tools). From my experience, I get about 3.5x speedup over PyPy when compiling, meaning 140x faster than python. PyPy is available for Python 3x and 2x code and again if you use an IDE like PyCharm you can interchange between say PyPy, Cython, and Python very easily (takes a little of initial learning and setup though).

Some people may argue with me on this one, but I find PyPy to be faster than Cython. But they're both great choices though.

Edit: I'd like to make another quick note about compiling: when you compile, the resulting binary is much bigger than your python script as it builds all dependencies into it, etc. But then you get a few distinct benefits: speed!, now the app will work on any machine (depending on which OS you compiled for, if not all. lol) without Python or libraries, it also obfuscates your code and is technically 'production' ready (to a degree). Some compilers also generate C code, which I haven't really looked at or seen if it's useful or just gibberish. Good luck.

Hope that helps.

jacktrader
  • 422
  • 4
  • 8
  • 2
    I know this is an older comment, but thanks! – kfrncs Mar 29 '19 at 22:03
  • No problem, I'm glad it was useful. – jacktrader Apr 26 '19 at 22:14
  • What software do you use to compile from PyPy interpretation? – Vasyl Vaskivskyi Oct 09 '19 at 15:45
  • Not specifically PyPy, just .py scripts. Nuitka if you want "C/C++ executable's, or C/C++ source code" and PyInstaller if you just want an executable (easier). There's also py2exe but I had less success with it, though I'm sure things have improved. PyInstaller is also cross platform, not just for Windows executable's (works with Linux, and Mac). Nuitka is unique because I think it's the only "compiler" that gives you usable source code back that you could in theory optimize further. There are a few others like bbFreeze, cx_Freeze, and py2app but I haven't tried them. Best of luck! – jacktrader Oct 09 '19 at 17:30
  • 1
    I have also found PyPy to run faster than Cython. In one test I actually found PyPy to be the same speed as a C++ version of the program (insertion sort). – Nv7 May 17 '20 at 03:11
8

Another option - to convert to C++ besides Shed Skin - is Pythran.

To quote High Performance Python by Micha Gorelick and Ian Ozsvald:

Pythran is a Python-to-C++ compiler for a subset of Python that includes partial numpy support. It acts a little like Numba and Cython—you annotate a function’s arguments, and then it takes over with further type annotation and code specialization. It takes advantage of vectorization possibilities and of OpenMP-based parallelization possibilities. It runs using Python 2.7 only.

One very interesting feature of Pythran is that it will attempt to automatically spot parallelization opportunities (e.g., if you’re using a map), and turn this into parallel code without requiring extra effort from you. You can also specify parallel sections using pragma omp > directives; in this respect, it feels very similar to Cython’s OpenMP support.

Behind the scenes, Pythran will take both normal Python and numpy code and attempt to aggressively compile them into very fast C++—even faster than the results of Cython.

You should note that this project is young, and you may encounter bugs; you should also note that the development team are very friendly and tend to fix bugs in a matter of hours.

boardrider
  • 4,470
  • 5
  • 39
  • 65
  • 1
    Note that "until 0.9.5 (included), Pythran was supporting Python 3 and Python 2.7. It now only supports Python 3." – Make42 Aug 24 '20 at 17:12
5

http://code.google.com/p/py2c/ looks like a possibility - they also mention on their site: Cython, Shedskin and RPython and confirm that they are converting Python code to pure C/C++ which is much faster than C/C++ riddled with Python API calls. Note: I haven’t tried it but I am going to..

ashley
  • 1,385
  • 1
  • 12
  • 18
5

I realize that an answer on a quite new solution is missing. If Numpy is used in the code, I would advice to try Pythran:

http://pythran.readthedocs.io/

For the functions I tried, Pythran gives extremely good results. The resulting functions are as fast as well written Fortran code (or only slightly slower) and a little bit faster than the (quite optimized) Cython solution.

The advantage compared to Cython is that you just have to use Pythran on the Python function optimized for Numpy, meaning that you do not have to expand the loops and add types for all variables in the loop. Pythran takes its time to analyse the code so it understands the operations on numpy.ndarray.

It is also a huge advantage compared to Numba or other projects based on just-in-time compilation for which (to my knowledge), you have to expand the loops to be really efficient. And then the code with the loops becomes very very inefficient using only CPython and Numpy...

A drawback of Pythran: no classes! But since only the functions that really need to be optimized have to be compiled, it is not very annoying.

Another point: Pythran supports well (and very easily) OpenMP parallelism. But I don't think mpi4py is supported...

serge-sans-paille
  • 2,011
  • 8
  • 9
paugier
  • 1,617
  • 2
  • 16
  • 36