How to handle C extensions for python apps with pip?

Question

For python applications that install with pip, how can you handle their C extension requirements automatically?

For example, the mysqlclient module requires development libs of MySQL installed on the system. When you initially install an application requiring that module it'll fail if the MySQL development libraries are not on the system. So the question is how do I solve this?

1. Is there a way to solve this with setup.py already that I do not know about?
2. If not am I supposed to use a pure python module implementation?

Note; I'm not looking for answers like "just use py2exe".

Answer 1

No. There is no way of including totally separate C library as a part of your build process unless you are writing an extension. Even in that case, you'll need to specify all .c files in ext_modules so that they all can be compiled as a part of your build process, which I know is not what you want.

The only thing you can do is to simply stop the build process and give user a reasonable error if mysql-devel (or libmysqlclient-dev) has not yet been installed.

One way to know if mysql-dev is installed is writing a simple C function that imports mysql.h and check if it is compiled successfully.

Note: mysql.h and my_global.h is part of libmysqlclient-dev package.

---

test/test_mysqlclient.c

// Taken from: http://zetcode.com/db/mysqlc

#include <my_global.h>
#include <mysql.h>

int main(int argc, char **argv)
{
  printf("MySQL client version: %s\n", mysql_get_client_info());
  exit(0);
}

---

Secondly, let's update our setup.py file so that it will be included as a part of the build process.

setup.py

#!/usr/bin/env python

import os
import subprocess

from setuptools import setup, Extension

def mysql_test_extension():
    process = subprocess.Popen(['which', 'mysql_config'],
                               stdout=subprocess.PIPE,
                               stderr=subprocess.PIPE,
                               close_fds=True)

    result, error = process.communicate()
    if process.returncode > 0:
        raise RuntimeError(error)

    config_command = result.strip()

    cflags = subprocess.check_output([config_command, '--cflags'], close_fds=True).strip()

    include_dirs = []
    extra_compile_args = []
    for arg in cflags.split(' '):
        if not arg.strip():
            continue
        elif arg.startswith('-I'):
            include_dirs.append(arg[2:])
        elif arg.startswith('-'):
            extra_compile_args.append(arg)
        else:
            extra_compile_args[-1] = extra_compile_args[-1] + ' ' + arg

    libs = subprocess.check_output([config_command, '--libs'], close_fds=True).strip()

    libraries = []
    linkers = []
    for arg in libs.split(' '):
        if not arg.strip():
            continue
        elif arg.startswith('-L'):
            libraries.append(arg[2:])
        elif arg.startswith('-'):
            linkers.append(arg)
        else:
            linkers[-1] = extra_compile_args[-1] + ' ' + arg

    return Extension('test_mysqlclient', ['test/test_mysqlclient.c'],
                     include_dirs=include_dirs,
                     library_dirs=libraries,
                     extra_link_args=linkers,
                     extra_compile_args=extra_compile_args)



setup(name='python-project',
      version='1.0',
      description='Python Project',
      classifiers=[
          'Development Status :: 5 - Production/Stable',
          'Environment :: Console',
          'Intended Audience :: Developers',
          'License :: OSI Approved :: MIT License',
          'Operating System :: OS Independent',
          'Programming Language :: Python :: 2.7',
          'Natural Language :: English',
      ],
      keywords='mysql python project',
      author='Ozgur Vatansever',
      url='http://github.com/ozgur/python-project/',
      license='MIT',
      packages=['some_project'],
      ext_modules = [mysql_test_extension()]
)

---

You can start building your package along with the test_mysqlclient file:

$ python setup.py build

If mysql-devel is not installed on your system, you'll get an build error similar to this:

test/test_mysqlclient.c:3:10: fatal error: 'my_global.h' file not found
#include <my_global.h>
     ^
1 error generated.

Answer 2

So the question is how do I solve this?

You have not solve this problem anyhow. There is no any method to describe external dependencies outside of python ecosystem in setup.py. Just provide it in a README.