Apache unable to load virtualenv properly: "ImportError: No module named 'psycopg2._psycopg'"

Question

I moved my Django project from Python 2.7 to Python 3.6.

Since then, I can't get my project up & running, as Apache complains as follows (full output after a restart):

[mpm_prefork:notice] [pid 2340] AH00173: SIGHUP received.  Attempting to restart
[:notice] [pid 25019] FastCGI: process manager initialized (pid 25019)
[:warn] [pid 2340] mod_wsgi: Compiled for Python/3.4.0.
[:warn] [pid 2340] mod_wsgi: Runtime using Python/3.4.3.
[mpm_prefork:notice] [pid 2340] AH00163: Apache/2.4.7 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141 PHP/5.5.9-1ubuntu4.21 OpenSSL/1.0.1f mod_wsgi/3.4 Python/3.4.3 configured -- resuming normal operations
[core:notice] [pid 2340] AH00094: Command line: '/usr/sbin/apache2'
[:error] [pid 25025] [client 10.0.10.117:49933] mod_wsgi (pid=25025): Target WSGI script '/home/myuser/projects/myproject/myproject_project/wsgi.py' cannot be loaded as Python module.
[:error] [pid 25025] [client 10.0.10.117:49933] mod_wsgi (pid=25025): Exception occurred processing WSGI script '/home/myuser/projects/myproject/myproject_project/wsgi.py'.
[:error] [pid 25025] [client 10.0.10.117:49933] Traceback (most recent call last):
[:error] [pid 25025] [client 10.0.10.117:49933]   File "/home/myuser/projects/myproject/myproject_project/wsgi.py", line 29, in <module>
[:error] [pid 25025] [client 10.0.10.117:49933]     application = get_wsgi_application()
[:error] [pid 25025] [client 10.0.10.117:49933]   File "/home/myuser/.virtualenvs/myproject-3.6/lib/python3.6/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
[:error] [pid 25025] [client 10.0.10.117:49933]     django.setup(set_prefix=False)
[:error] [pid 25025] [client 10.0.10.117:49933]   File "/home/myuser/.virtualenvs/myproject-3.6/lib/python3.6/site-packages/django/__init__.py", line 27, in setup
[:error] [pid 25025] [client 10.0.10.117:49933]     apps.populate(settings.INSTALLED_APPS)
[:error] [pid 25025] [client 10.0.10.117:49933]   File "/home/myuser/.virtualenvs/myproject-3.6/lib/python3.6/site-packages/django/apps/registry.py", line 85, in populate
[:error] [pid 25025] [client 10.0.10.117:49933]     app_config = AppConfig.create(entry)
[:error] [pid 25025] [client 10.0.10.117:49933]   File "/home/myuser/.virtualenvs/myproject-3.6/lib/python3.6/site-packages/django/apps/config.py", line 116, in create
[:error] [pid 25025] [client 10.0.10.117:49933]     mod = import_module(mod_path)
[:error] [pid 25025] [client 10.0.10.117:49933]   File "/usr/lib/python3.4/importlib/__init__.py", line 109, in import_module
[:error] [pid 25025] [client 10.0.10.117:49933]     return _bootstrap._gcd_import(name[level:], package, level)
[:error] [pid 25025] [client 10.0.10.117:49933]   File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
[:error] [pid 25025] [client 10.0.10.117:49933]   File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
[:error] [pid 25025] [client 10.0.10.117:49933]   File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
[:error] [pid 25025] [client 10.0.10.117:49933]   File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
[:error] [pid 25025] [client 10.0.10.117:49933]   File "<frozen importlib._bootstrap>", line 1129, in _exec
[:error] [pid 25025] [client 10.0.10.117:49933]   File "<frozen importlib._bootstrap>", line 1471, in exec_module
[:error] [pid 25025] [client 10.0.10.117:49933]   File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
[:error] [pid 25025] [client 10.0.10.117:49933]   File "/home/myuser/.virtualenvs/myproject-3.6/lib/python3.6/site-packages/django/contrib/postgres/apps.py", line 7, in <module>
[:error] [pid 25025] [client 10.0.10.117:49933]     from .signals import register_hstore_handler
[:error] [pid 25025] [client 10.0.10.117:49933]   File "/home/myuser/.virtualenvs/myproject-3.6/lib/python3.6/site-packages/django/contrib/postgres/signals.py", line 1, in <module>
[:error] [pid 25025] [client 10.0.10.117:49933]     from psycopg2 import ProgrammingError
[:error] [pid 25025] [client 10.0.10.117:49933]   File "/home/myuser/.virtualenvs/myproject-3.6/lib/python3.6/site-packages/psycopg2/__init__.py", line 50, in <module>
[:error] [pid 25025] [client 10.0.10.117:49933]     from psycopg2._psycopg import (                     # noqa
[:error] [pid 25025] [client 10.0.10.117:49933] ImportError: No module named 'psycopg2._psycopg'

However when I try to load the problem module (psycopg2) in the virtualenv manually, it is perfectly possible to do so:

(myproject-3.6)myuser@Server:$ python
Python 3.6.0 (default, Jan 13 2017, 00:00:00)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2
>>>

I'm using WSGI. Its file content is as follows:

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "my_project.settings")
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

My main Apache2 site config is as follows (there is a section for the admin/ as well):

WSGIApplicationGroup %{GLOBAL}
WSGIScriptAlias / /home/myuser/projects/myproject/myproject_project/wsgi.py
#WSGIPythonPath /home/myuser/projects/myproject:/home/myuser/.virtualenvs/myproject/local/lib/python2.7/site-packages
WSGIPythonPath /home/myuser/projects/myproject:/home/myuser/.virtualenvs/myproject-3.6/lib/python3.6/site-packages
<Directory /home/myuser/projects/myproject/myproject_project>
        SSLRequireSSL
        <Files wsgi.py>
                Order deny,allow
                Require all granted
        </Files>
</Directory>

I've installed the python3-psycopg2 package through apt-get. And of course my virtualenv has psycopg2 installed as well:

(myproject-3.6)myuser@Server:~/projects/$ pip freeze | grep psy
psycopg2==2.7

What is going wrong?

I'm using Ubuntu 14.04 LTS.

Please read warnings in documentation at http://modwsgi.readthedocs.io/en/develop/user-guides/virtual-environments.html where it says you cannot force mod_wsgi compiled for one Python version to use a Python virtual environment based off a different Python version. In other words, you must install mod_wsgi for Python 3.6. Your mod_wsgi is built for Python 3.4. That documentation also tells you how to correctly specify what virtual environment to use. How you do it is not the recommended way. — Graham Dumpleton, Mar 04 '17 at 22:30

score 1 · Accepted Answer · answered Sep 15 '17 at 05:19

I had exactly the same problem trying to run my Django 1.11 application using Python 3.6, Apache2.4 with WSGI on my Debian 8 machine.

The problem is libapache2-mod-wsgi-py is compiled for a specific versions of both apache2 and python and it is not compatible with different python version. Of course,

libapache2-mod-wsgi-py is used for python 2.7,
libapache2-mod-wsgi-py3 --- for python3,

but minor python version is also matters.

Let's see. Inside @SaeX stacktrace the most of lines are about your Python 3.6 venv such

...
File "/home/myuser/.virtualenvs/myproject-3.6/lib/python3.6/site-packages/django/apps/config.py", line 116, in create
...

but there is a line with Python 3.4 call

...
File "/usr/lib/python3.4/importlib/__init__.py", line 109, in import_module
...

I assume that in his OS repositories (that time) the latest python3 version available from apt-get install python3 was 3.4.n and the result of apt-get install libapache2-mod-wsgi-py3 is mod_wsgi compiled for Python 3.4. That's the reason.

In my situation I firstly caught python2.7 in a stacktrace

File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module'

that meant I try to use libapache2-mod-wsgi-py instead of libapache2-mod-wsgi-py3. But when I installed libapache2-mod-wsgi-py3 I've get python3.5 in the stacktrace

File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module'

I've solved the problem in two ways, both were okay for me:

use the latest available python3 version from package repositories of your OS;
build mod-wsgi for python3.6

1. Latest system Python 3

@SaeX used this way:

Thanks; downgrading my venv to Python 3.4 fixed the issue

In your case latest Python 3 version could differ.

apt-get update
apt-get install python3 python3-venv

# check python3 version
python3 -V
# >>> Python 3.5.4 is for my Debian 8 in September 2017

# setting virtual environment for WSGI
python3 -m venv /home/pyvenv_354
source /home/pyvenv_354/bin/activate
# check venv and get path for WSGI
(pyvenv_354) python -c 'import sys; print(sys.prefix)'
# >>> /home/pyvenv_354

# mod_wsgi from repo must be compatible
apt install libapache2-mod-wsgi-py3

And now this python3.5 venv could be used in apache2

# sites-available/mysite.conf
...
Define WWW_ROOT /var/www/mysite/mysite
...
Define PYTHON_VENV_HOME /home/pyvenv_354
Define PYTHON_VENV_PACKAGES ${PYTHON_VENV_HOME}/lib/python3.5/site-packages
WSGIDaemonProcess mysite processes=2 python-path=${WWW_ROOT}:${PYTHON_VENV_PACKAGES}
WSGIProcessGroup mysite
WSGIScriptAlias / ${MAIN_APP_PATH}/wsgi.py

And finally restart Apache2 using your favourite way, e.g.

service apache2 restart

2. Build mod_wsgi for Python 3.6

I've used mod_wsgi quick installation guide to perform it. Note: apache2-dev and python-dev packages are required.

apt-get install python3.6 python3.6-dev apache2-dev
# checking Python version
python3.6 -V
# >>> 3.6.2

# set virtual environment
python3.6 -m venv /home/pyvenv_362

# download mod_wsgi sources from https://github.com/GrahamDumpleton/mod_wsgi/releases
# x, y, z --- version, select using link above
cd /usr/src/
wget https://github.com/GrahamDumpleton/mod_wsgi/archive/x.y.z.tar.gz
tar xvfz x.y.z.tar.gz
cd mod_wsgi-x.y.z

# set python3.6 venv for mod_wsgi
./configure --with-python=/home/pyvenv_362/bin/python

make
make install

# >>> Libraries have been installed in:
#   /usr/lib/apache2/modules
#   mod_wsgi.so

# adding module to apache2
cat /etc/apache2/mods-enabled/wsgi.load
nano /etc/apache2/httpd.conf
add to httpd.conf line `LoadModule wsgi_module modules/mod_wsgi.so`

And now python3.6 venv could be used in apache2

# sites-available/mysite.conf
...
Define WWW_ROOT /var/www/mysite/mysite
...
Define PYTHON_VENV_HOME /home/pyvenv_362
Define PYTHON_VENV_PACKAGES ${PYTHON_VENV_HOME}/lib/python3.6/site-packages
WSGIDaemonProcess mysite processes=2 python-path=${WWW_ROOT}:${PYTHON_VENV_PACKAGES}
WSGIProcessGroup mysite
WSGIScriptAlias / ${MAIN_APP_PATH}/wsgi.py

And finally restart Apache2 using your favourite way, e.g.

service apache2 restart

Apache unable to load virtualenv properly: "ImportError: No module named 'psycopg2._psycopg'"

1 Answers1