25

Problem description:

Using pip, I upgraded to the latest version of requests (version 2.7.0, with pip show requests giving the location /usr/local/lib/python2.7/dist-packages). When I import requests and print requests.__version__ in the interactive command line, though, I am seeing version 2.2.1. It turns out that Python is using the pre-installed Ubuntu version of requests (requests.__file__ is /usr/lib/python2.7/dist-packages/requests/__init__.pyc -- not /user/local/lib/...).

From my investigation, this fact is caused by Ubuntu's changes to the Python search path (I run Ubuntu 14.04) by prepending the path to Ubuntu's Python package (for my machine, this happens in usr/local/lib/python2.7/dist-packages/easy-install.pth). In my case, this causes the apt-get version of requests, which is pre-packaged with Ubuntu, to be used, rather than the pip version I want to use.

What I'm looking for:

I want to globally prepend pip's installation directory path to Python's search path (sys.path), before the path to Ubuntu's Python installation directory. Since requests (and many other packages) are used in many Python scripts of mine, I don't want to manually change the search path for every single file on my machine.

Unsatisfactory Solution 1: Using virtualenv

Using virtualenv would cause an unnecessary amount of change to my machine, since I would have to reinstall every package that exists globally. I only want to upgrade from Ubuntu's packages to pip's packages.

Unsatisfactory Solution 2: Changing easy-install.pth

Since easy-install.pth is overwritten every time easy-install is used, my changes to easy-install.pth would be removed if a new package is installed. This problem makes it difficult to maintain the packages on my machine.

Unsatisfactory (but best one I have so far) Solution 3: Adding a separate .pth file

In the same directory as easy-install.pth I added a zzz.pth with contents:

import sys; sys.__plen = len(sys.path)
/usr/lib/python2.7/dist-packages/test_dir
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

This file is read by site.py when Python is starting. Since its file name comes after easy-install.pth alphanumerically, it is consumed by site.py afterwards. Taken together, the first and last lines of the file prepend the path to sys.path (these lines were taken from easy-install.pth).

I don't like how this solution depends on the alphanumeric ordering of the file name to correctly place the new path.

PYTHONPATHs come after Ubuntu's paths

Another answer on Stack Overflow didn't work for me. My PYTHONPATH paths come after the paths in easy-install.pth, which uses the same code I mention in "Unsatisfactory solution 3" to prepend its paths.

Thank you in advance!

Community
  • 1
  • 1
max
  • 259
  • 1
  • 3
  • 3

8 Answers8

14

This is not recommended, as it hard-codes a path and makes it difficult to run the script elsewhere, but you can do

>>> import sys
>>> sys.path.insert(0,'/home/anand/')
>>> print(sys.path)
['/home/anand/', '', '/usr/local/lib/python2.7/dist-packages/_pdbpp_path_hack', '/usr/local/lib/python2.7/dist-packages/goose-0.0.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/jieba-0.33-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/cssselect-0.9.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/nanoservice-0.1.5-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/nanomsg-1.0a2-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/msgpack_python-0.4.2-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/DecisionTree-2.2.5-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/nudepy-0.2-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/wsgilog-0.3-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/distribute-0.7.3-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/PIL-1.1.7-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/munkres-1.0.7-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/parsedatetime-1.4-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/argparse-1.3.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/tusker-0.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/SQLAlchemy-1.0.3-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/numpy-1.9.2-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/turkic-0.2.5-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/scikits.bootstrap-0.3.2-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/pyvision-0.1-py2.7-linux-x86_64.egg', '/home/anand/playspace/languages/python_pkgs/ets', '/usr/local/lib/python2.7/dist-packages/Scrapy-1.1.0dev1-py2.7.egg', '/usr/lib/python2.7/dist-packages', '/home/anand/playspace', '/home/anand/workspace/pyvision/src', '/home/anand/playspace/yapf', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/Orange/orng', '/usr/local/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/wx-3.0-gtk2']
>>>

After this your imports will look into the prepended path before looking anywhere else.

Patrick Sanan
  • 1,900
  • 1
  • 20
  • 25
  • _"This is not recommended..."_ Then why are you recommending it? What's the recommended way then? – code_dredd Mar 04 '19 at 22:23
  • I didn't mean this is not the recommended way. I mean, prepending custom paths to python path is not recommended in 99.9% of cases. There's almost no good reason to do this. The OP's question was a special case, and probably not relevant anymore with later ubuntu versions. – Nandhini Anand Jeyahar Mar 06 '19 at 10:55
  • 3
    Example use case: I've clone from GitHub the source of a Python package I want to explore (e.g. insert break points, print statements, test modifications). I already have this package installed, and I don't want to uninstall that, install my tests, etc. I can use this technique in a test script to change the package copy specifically without impacting anything else. It's useful here, but definitely not as a general way to run code. – Demitri May 05 '20 at 21:31
7

You shouldn't need to mess with pip's path, python actually handles it's pathing automatically in my experience. It appears you have two pythons installed. If you type:

which pip
which python

what paths do you see? If they're not in the same /bin folder, then that's your problem. I'm guessing that the python you're running (probably the original system one), doesn't have it's own pip installed. You probably just need make sure the path for the python you want to run should come before /usr/bin in your .bashrc or .zshrc

If this is correct, then you should see that:

which easy_install

shares the same path as the python installation you're using, maybe under /usr/local/bin. Then just run:

easy_install pip

And start installing the right packages for the python that you're using.

fivetentaylor
  • 1,188
  • 6
  • 11
  • I was trying to change the search path which my Python binary uses on startup. No matter the binary I use, the rules for determining a search path should be the same. – max Jul 23 '15 at 22:29
  • 1
    Hey @max, I understand, but that is dangerous if you have different versions of python installed. You should have the proper pip installed alongside each python, and then the proper packages will be installed for that python. If you don't then you'll have your python pointing to wrong versions of things and that'd be bad. – fivetentaylor Jul 24 '15 at 15:31
  • I have to agree with @fivetentaylor here. I had a project that I had to use Python 3.X on, and when you install that version you get a pip for that version of Python. There is a `python, python2, python2.7, python3, python3.4` and then there are `pip, pip2, pip2.7, pip3, pip3.4`. The 2's installs to the `dist-packages` for `Py2` and the 3's install to the one for `Py3`. If you want to install a package for Python 3.4, then you either you do `python3.4 -m pip install ` or `pip3.4 install ' and it will install in 3.4's `dist-packages`. – AMR Jul 26 '15 at 02:12
  • The interpreter for whatever version of python you are running will import from it's version of `dist-packages` and you will get the correct one. I haven't personally used virtualenv, but from what I have read, it seems to be the way Python developers suggest going for multi-python dev environments. – AMR Jul 26 '15 at 02:16
  • This won't fix the problem, `easy_install` and `pip` version their commands in the same way. You'll keep running into problems unless you manage your `PYTHONPATH` and `PATH` using virtualenvs. – James Tocknell Aug 02 '15 at 16:06
  • I don't think so, the problem is something like calling python at: /usr/bin/python and calling pip at /bin/pip. This would happen if /usr/bin came first on PATH, but there was no pip installed there. As long as the pip you're calling lives a the same path as your python, then pip will install in the right place. – fivetentaylor Aug 02 '15 at 16:18
4

Answer to direct question

You could create a directory called sitecustomize in your site-packages directory. We'll turn this into a sitecustomize module as described here (Python 2 here). Specifically:

an attempt is made to import a module named sitecustomize, which can perform arbitrary site-specific customizations. It is typically created by a system administrator in your site-packages directory.

In the sitecustomize directory create a file called __init__.py and add the manipulations you want to perform there. A very simple example is:

import sys
sys.path = ['/your/path/to/pip/install'] + sys.path

In your case, I think your/path... would be /usr/local/lib/python2.7/dist-packages. You might want to do something more sophisticated, but this crudely prepends to sys.path and is run whenever python is started (e.g. starting the interpreter in the command line, or running a python script from a file).

Caveat

I'm not a huge advocate of doing this - it's a bit of a blunt way to do what you want. But you specifically say that using a virtualenv is undesirable for you and you want to make the change "globally" and I think this will do what you want.

Thoughts on underlying issue

I think @fivetentaylor's answer is on the right track here - it appears you are using pip from one install with python executable for another. Masking this by messing with the path could get very confusing very quickly. I'd definitely ensure you have a separate pip for each install of python and you use that. That should keep the directory structures for the separate installs separate. Otherwise, you are forcing one install to use packages from a different installation's directories. No problem technically, but confusing logistically.

Community
  • 1
  • 1
J Richard Snape
  • 18,946
  • 5
  • 47
  • 73
3

Using virtualenv would cause an unnecessary amount of change to my machine, since I would have to reinstall every package that exists globally. I only want to upgrade from Ubuntu's packages to pip's packages.

Nope, you can use --system-site-packages.

Edit

# make your new virtualenv
user@darkstar:~$ mkvirtualenv --system-site-packages max
(max)user@darkstar:~$ python
>>> pprint(sys.path)
['',
 '/home/user/.virtualenvs/max/lib64/python27.zip',
 '/home/user/.virtualenvs/max/lib64/python2.7',
 '/home/user/.virtualenvs/max/lib64/python2.7/plat-linux2',
 '/home/user/.virtualenvs/max/lib64/python2.7/lib-tk',
 '/home/user/.virtualenvs/max/lib64/python2.7/lib-old',
 '/home/user/.virtualenvs/max/lib64/python2.7/lib-dynload',
 '/usr/lib64/python2.7',
 '/usr/lib/python2.7',
 '/usr/lib64/python2.7/lib-tk',
 '/home/user/.virtualenvs/max/lib/python2.7/site-packages',
 '/usr/lib64/python2.7/site-packages/google_api_python_client-1.2-py2.7.egg',
 '/usr/lib64/python2.7/site-packages',
 '/usr/lib64/python2.7/site-packages/PIL',
 '/usr/lib64/python2.7/site-packages/gtk-2.0',
 '/usr/lib64/python2.7/site-packages/IPython/extensions']

As you see, the path of this virtualenv include the system path. To check if it was working, i installed a package system-wide after making the virtualenv.

root@darkstar:~: pip install igraph
Collecting igraph
  Downloading igraph-0.1.8-py2.py3-none-any.whl (119kB)
    100% |████████████████████████████████| 122kB 1.7MB/s
Collecting ipython (from igraph)
  Downloading ipython-3.2.1-py2-none-any.whl (3.4MB)
    100% |████████████████████████████████| 3.4MB 203kB/s 
Installing collected packages: ipython, igraph
Successfully installed igraph-0.1.8 ipython-3.2.1
root@darkstar:~: python -c 'print __import__("igraph")'
<module 'igraph' from '/usr/lib64/python2.7/site-packages/igraph/__init__.pyc'>

(max)user@darkstar:max$ python -c 'print __import__("igraph")'
<module 'igraph' from '/usr/lib64/python2.7/site-packages/igraph/__init__.pyc'>

Obviously, what is installed inside the virtualenv takes precedence over the system-wide libraries.

I belive that answer your needs.

bufh
  • 2,647
  • 24
  • 32
2

Well, the alternatives presented by the others are very acceptable and might even be better. However, if you are intent on using the sys.path() way, then just treat it like a list and use the insert method.

import sys
sys.path.insert(0, "path_to_pip") 
from subprocess import call
call("sudo pip install requests") 
ytpillai
  • 3,194
  • 25
  • 43
2

I would do this with sitecustomize as described in the site.py docs. This file is imported after the initial sys.path is configured and you can use it to alter sys.path in arbitrary ways as needed.

I've used it as a sysadmin to include custom release locations and it does the job quite nicely.

https://docs.python.org/2/library/site.html

matthewatabet
  • 1,385
  • 10
  • 22
1

While bufh's answer will solve your problem now, you'll likely find that there'll be some other package where you don't want to use the Ubuntu-provided version. So here's why you want to use virtualenvs to manage the versions of packages (and not try to override the system versions).

As you've noticed, the order of sys.path sets the order that python packages are found. This means that changing sys.path affects how python scripts find their imports, both scripts you've written, and those provided by Ubuntu. Given that python scripts are used in Ubuntu programs, it's possible to "break" Ubuntu in interesting ways by changing which version of python packages Ubuntu programs use (which is the reason dist-packages exists).

To avoid this, virtualenv was created, which effectively allows there to be different sets of packages to be used. There's now a bunch of utilities which make using and managing virtualenvs easier. The one that's probably of most interest for you is pipsi, which creates a virtualenv per script, and avoids the need to activate it.

James Tocknell
  • 447
  • 4
  • 14
1

From the sounds of things, Ubuntu is using a package path configuration file as documented here to set up any packages it installs.

Looking at site.py I see that there is a specific order of path resolution that invokes the configuration files as it resolves the site packages directories.

I think that gives you three options that I can see:

  1. Use virtualenv --system-site-packages as per @bufh's answer.
  2. Use pip user installs to set up the packages you need in the path before the standard site packages.
  3. Use sitecustomize to re-write your sys.path (e.g. to put your local directories first).
Peter Brittain
  • 12,717
  • 3
  • 34
  • 49