Django: No module named 'psycopg2' when launching migrations to heroku

Question

I am new to django and I created a website and now I am in the deployment phase, all is well run until launching migrating from the database to the server. I'll show you everything I've done since the beginning

structure of my project

OC_project8/
    pur_beurre/
        catalog/
            dumps/
                catalog.json
            ...        # other classic file
        media/
        pur_beurre/
            static/
                .gitignore
            ...        # other classic file
        static/
        templates/
        users/
        manage.py
        Procfile
    venv/
    requirements.txt/
        certifi==2019.9.11
        chardet==3.0.4
        coverage==4.5.4
        dj-database-url==0.5.0
        Django==2.2.6
        django-debug-toolbar==2.0
        gunicorn==19.9.0
        idna==2.8
        mysqlclient==1.4.4
        Pillow==6.2.0
        pytz==2019.3
        requests==2.22.0
        sqlparse==0.3.0
        urllib3==1.25.6
        whitenoise==4.1.4
    .gitignore/

All command was run in a virtual environement, for exemple:

(venv) ~/OC_project8/pur_beurre$ sudo apt-get install python-psycopg2

I did all the necessary steps before runing the command git push heroku master (git commit, heroku create my-app ...) if you need more detail on these steps tell me.

After runing the command git push heroku master I had this error:

....

remote:        Collecting pkg-resources==0.0.0 (from -r /tmp/build_cac68f54540915062647a425c52dc61b/requirements.txt (line 11)) 
remote:          Could not find a version that satisfies the requirement pkg-resources==0.0.0 (from -r/tmp/build_cac68f54540915062647a425c52dc61b/requirements.txt (line 11)) (from versions: )
remote:        No matching distribution found for pkg-resources==0.0.0 (from -r /tmp/build_cac68f54540915062647a425c52dc61b/requirements.txt (line 11))
remote:  !     Push rejected, failed to compile Python app.
remote: 
remote:  !     Push failed
remote: Verifying deploy...
remote: 
remote: !       Push rejected to pur-beurre-oc8.
remote: 
To https://git.heroku.com/pur-beurre-oc8.git
! [remote rejected] master -> master (pre-receive hook declined)
error: impossible de pousser des références vers 'https://git.heroku.com/pur-beurre-oc8.git'

I saw in stackoverflow that I have to do that:

pip uninstall pkg-resources==0.0.0

After I made a commit:

git add requirements.txt
git commit -m "uninstalling pkg-resources==0.0.0"

And now git push heroku master work fine, but when I access my website I had the error: Apllication error. I don't know if I'm getting wet but I think because I have not done the migration of the database yet.

So I run the command heroku run python pur_beurre/manage.py migrate

And now I had this error:

Running python pur_beurre/manage.py migrate on ⬢ pur-beurre-oc8... up, run.4138 (Free)
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 20, in <module>
    import psycopg2 as Database
ModuleNotFoundError: No module named 'psycopg2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pur_beurre/manage.py", line 21, in <module>
    main()
  File "pur_beurre/manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/app/.heroku/python/lib/python3.6/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/app/.heroku/python/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/app/.heroku/python/lib/python3.6/site-packages/django/contrib/auth/base_user.py", line 47, in <module>
    class AbstractBaseUser(models.Model):
  File "/app/.heroku/python/lib/python3.6/site-packages/django/db/models/base.py", line 117, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/app/.heroku/python/lib/python3.6/site-packages/django/db/models/base.py", line 321, in add_to_class
    value.contribute_to_class(cls, name)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/db/models/options.py", line 204, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/app/.heroku/python/lib/python3.6/site-packages/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/db/utils.py", line 201, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/app/.heroku/python/lib/python3.6/site-packages/django/db/utils.py", line 110, in load_backend
    return import_module('%s.base' % backend_name)
  File "/app/.heroku/python/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/app/.heroku/python/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 24, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2'

I don't know why I have this error since psycopg2 is a library required to use PostgreSQL and I am using MySQL, here is my settings.py:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',   
        'NAME': 'openfoodfacts',             
        'USER': 'root',
        'PASSWORD': 'my-password',        
        'HOST': '',                    
        'PORT': '5432',                         
    }
}

In any case after having this error I tried to install psycopg2:

pip install psycopg2

But I had another error:

Collecting psycopg2
  Downloading https://files.pythonhosted.org/ packages/84/d7/6a93c99b5ba4d4d22daa3928b983cec66df4 536ca50b22ce5dcac65e4e71/psycopg2-2.8.4.tar.gz (377kB)
    100% |████████████████████████████████| 378kB 1.8MB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/rouizi/OC_project8/venv/lib/python3.6/site-packages/setuptools/__init__.py", line 12, in <module>
        import setuptools.version
      File "/home/rouizi/OC_project8/venv/lib/python3.6/site-packages/setuptools/version.py", line 1, in <module>
        import pkg_resources
    ModuleNotFoundError: No module named 'pkg_resources'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-gf4fxs4k/psycopg2/

Maybe I got this error because I uninstalled in the beginning pkg-resources ! But I tried to reinstall pkg-resources:

sudo apt-get install --reinstall python-pkg-resources

And I rerun the command pip install psycopg2, but still have the same error

By searching on google I found here No module named pkg_resources that it was necessary to run this command:

wget https://bootstrap.pypa.io/ez_setup.py -O - | python

And I rerun the command pip install psycopg2, but I have another error:

Collecting psycopg2
  Using cached https://files.pythonhosted.org/ packages/84/d7/6a93c99b5ba4d4d22daa3928b983cec66df 4536ca50b22ce5dcac65e4e71/psycopg2-2.8.4.tar.gz
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/psycopg2.egg-info
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).


    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-sglmdb2o/psycopg2/

And if I run the command which pg_config it return nothing, but after some research I fond here pg_config executable not found that it was necessary to run this command:

sudo apt-get install libpq-dev

and now which pg_config return this:

/usr/bin/pg_config

So I rerun pip install psycopg2 and another error occured:

Collecting psycopg2
  Using cached https://files.pythonhosted.org/ packages/84/d7/6a93c99b5ba4d4d22 daa3928b983cec66df4536ca50b22ce5dcac65e4e71/psycopg2-2.8.4.tar.gz
Building wheels for collected packages: psycopg2
  Running setup.py bdist_wheel for psycopg2 ... error
  Complete output from command /home/rouizi/OC_project8/venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-lu4ihs81/psycopg2/setup.py';f=getattr(tokenize,    
'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmphgjf714ppip-wheel- --python-tag cp36:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for psycopg2
  Running setup.py clean for psycopg2
Failed to build psycopg2
Installing collected packages: psycopg2
  Running setup.py install for psycopg2 ... error



           #I cut here because it's very long


    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-lu4ihs81/psycopg2/setup.py", line 611, in <module>
        'Documentation': 'http://initd.org/psycopg/docs/',
      File "/home/rouizi/OC_project8/venv/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/rouizi/OC_project8/venv/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.6/distutils/command/install.py", line 601, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/rouizi/OC_project8/venv/lib/python3.6/site-packages/setuptools/command/install_scripts.py", line 17, in run
        import setuptools.command.easy_install as ei
      File "/home/rouizi/OC_project8/venv/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 47, in <module>
        from setuptools.sandbox import run_setup
      File "/home/rouizi/OC_project8/venv/lib/python3.6/site-packages/setuptools/sandbox.py", line 15, in <module>
        import pkg_resources.py31compat
    ModuleNotFoundError: No module named 'pkg_resources.py31compat'

    ----------------------------------------
Command "/home/rouizi/OC_project8/venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-lu4ihs81/psycopg2/setup.py';f=getattr(tokenize, 'open', open)(__file__)
;code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-eya3xv3b-record/install-record.txt --single-version-externally-managed 
--compile --install-headers /home/rouizi/OC_project8/venv/include/site/python3.6/psycopg2" failed with error code 1 in /tmp/pip-build-lu4ihs81/psycopg2/

I tried this command:

sudo apt-get install --reinstall python-setuptools

And now when I try to run pip install psycopg2 he return to me:

Requirement already satisfied: psycopg2 in /home/rouizi/OC_project8/venv/lib/python3.6/site-packages

But if I try to run the command heroku run python pur_beurre/manage.py migrate I still have the error:

django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2'

But if I run sudo apt-get install python-psycopg2 It worked but I still have the error when runing heroku run python pur_beurre/manage.py migrate Since I still had the same error, so I tried uninstall psycopg2:

sudo pip uninstall psycopg2

But it return to me that:

The directory '/home/rouizi/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Not uninstalling psycopg2 at /usr/lib/python2.7/dist-packages, outside environment /usr

And there I am stuck for a day without finding a solution, I hope you can help

sorry for the long post and for my misspelling

This is just too much. I think you need to pinpoint some specific issue that you can think of and focus on that, rather than give the entire history. I could be wrong, but that's an awful lot of info to wade through — roganjosh, Oct 31 '19 at 20:18
Just so you know apt-get is agnostic of a virtual environment. When you run `sudo apt-get install python-psycopg2` it installs the python2 version to the system environment. You can verify this with `apt show psycopg2` command. What I think might help, with your virtual environment activated run `pip3 list | grep psycopg` see what that returns. If it returns nothing, try `pip3 install psycopg2-binary`. I've had issues before when I didn't use the binary. — R. Arctor, Oct 31 '19 at 20:25
@Arctor pip3 list | grep psycopg returns this : psycopg2 (2.8.4) and also this : DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning. — Yacine Rouizi, Oct 31 '19 at 21:01
Please [edit] your question and show us your `requirements.txt`. Also, please confirm that you don't have `Pipfile` and / or `Pipfile.lock` files. — Chris, Oct 31 '19 at 23:29

score 2 · Accepted Answer · answered Nov 01 '19 at 10:41

I solved my problem ! I ran the command pip install psycopg2, it return to me the error:

ModuleNotFoundError: No module named 'pkg_resources.py31compat'

But if I try to rerun the command pip install psycopg2, it says:

Requirement already satisfied: psycopg2 in /home/rouizi/OC_project8/venv/lib/python3.6/site-packages

So I tryied to commit my change:

pip freeze > requirements.txt
git add requirements.txt
git commit -m "installing psycopg2"
git push heroku master

And now heroku run python pur_beurre/manage.py migrate work fine.

I don't know why since I had an error, if someone has an explanation will be well

Django: No module named 'psycopg2' when launching migrations to heroku

structure of my project

1 Answers1