151

I've just started working with setuptools and virtualenv. My package requires the latest python-gearman that is only available from GitHub. The python-gearman version that's on PyPI is an old one. The Github source is setuptools-compatible, i.e. has setup.py, etc. Is there a way to make setuptools download and install the new version instead of looking for it on PyPI and installing the old one?

FYI, the new python-gearman is http://github.com/mtai/python-gearman

andrei
  • 1,893
  • 3
  • 16
  • 15
  • 2
    Is there a reason you're trying to install a python package directly off of Git instead of downloading the source from there and using `python setup.py install` in the source directory? – Andrew Aug 12 '10 at 22:21
  • 4
    I want my package to be deployed on multiple machines and all its dependencies installed automatically. – andrei Aug 12 '10 at 22:24
  • 2
    You can use `easy_install` or `pip` to install it straight from Github. But there's also another solution, have you considered adding the package to PyPI? – Wolph Aug 12 '10 at 22:32
  • 2
    Since it's simply for deployment, why not use `buildout`? It has a couple of ready-made Git plugins. – Wolph Aug 12 '10 at 22:33
  • solution here: http://stackoverflow.com/a/17442663/1841871 – zazabe Oct 29 '14 at 03:18

4 Answers4

161

The key is to tell easy_install where the package can be downloaded. In this particular case, it can be found at the url http://github.com/mtai/python-gearman/tarball/master. However, that link by itself won't work, because easy_install can't tell just by looking at the URL what it's going to get.

By changing it to http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta instead, easy_install will be able to identify the package name and its version.

The final step is to add the URL to your package's dependency_links, e.g.:

setup(
   ...
   dependency_links = ['http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta']
)

Now, when YOUR package is being installed, easy_install will discover that there is a "gearman 2.0.0beta" available for download from that URL, and happily pick it over the one on PyPI, if you specify "gearman>=2.0.0beta" in your dependencies..

(Normally, the way this sort of thing is done is to include a link on one's PyPI page to the downloadable source; in this case, if the author of the gearman package had included a link like the above, you'd be already set. Typically, people mark the development version with 'myproject-dev' and then people use a requirement of 'myproject>=somever,==dev', so that if there isn't a package of somever or higher, easy_install will try to check out or download the release.)

You'll need to specify --process-dependency-links when using pip. Note that dependency links processing has been deprecated and will be removed in a future release.

devin_s
  • 2,975
  • 1
  • 22
  • 32
PJ Eby
  • 8,196
  • 5
  • 21
  • 17
  • 1
    I did what you suggested, but when I run "python setup.py develop", it says "writing dependency_links to foo.egg-info/dependency_links.txt", but doesn't actually download and install the package. I'm using a setuptools-based virtualenv if that helps. – andrei Aug 18 '10 at 17:31
  • 17
    You need to also have install_requires='gearman>=2.0.0beta'; did you include that? – PJ Eby Aug 22 '10 at 23:11
  • That worked. But in general, would you suggest using something like buildout for enforcing such requirements or, perhaps, Puppet or Chef? – andrei Aug 24 '10 at 18:38
  • I hear lots of good things about buildout, but haven't used it myself yet. If you need other things besides just Python code installed, I'd recommend you at least investigate buildout. – PJ Eby Aug 25 '10 at 15:58
  • When using this method you may need to escape dashes in the name or version. See https://pythonhosted.org/setuptools/setuptools.html#dependencies-that-aren-t-in-pypi – danza Sep 05 '14 at 09:07
  • 4
    It doesn't work for me, with the `beta` suffix on an existing version on PyPI, it will still install the package from PyPI instead of the one defined in `dependency_links`. If you try to set a higher version than what exists on PyPI with `#egg=package-version`, the setup tool will complain with a `Could not find a version that satisfies the requirement` error and a list of all versions available on PyPI. Note that I'm trying to build my package with `sdist`, then installing it with `pip install http://url/to/my/generated/tar`. – zazabe Oct 28 '14 at 12:15
  • 1
    ok, by installing my package with `easy_install http://url/to/my/generated/tar`, everything works like expected... Any idea why ? – zazabe Oct 28 '14 at 12:24
  • It's weird that if your `install_requires` doesn't specify a version, the `dependency_link` must have a version tag and it must be numeric. Whereas if you do specify something like `-dev` for the version tag in `dependency_links` while also saying `==dev` for the version constraint, then it just works. It seems that setuptools thinks that with no version constraints, you're looking for a package with any numeric version but does not allow `dependency_links` with no versions or alphabetical versions. – CMCDragonkai Jul 20 '18 at 06:56
  • 5
    `--process-dependency-links` has been removed as of `pip` 19! See: https://github.com/pypa/pip/issues/6162 – phoenix Apr 04 '19 at 12:49
69

You can use the pip install protocol+location[@tag][#egg=Dependency] format to install directly from source using pip.

Git

pip install git+https://github.com/username/repo.git
pip install git+https://github.com/username/repo.git@MyTag
pip install git+https://github.com/username/repo.git@MyTag#egg=ProjectName

Mercurial

pip install hg+https://hg.myproject.org/MyProject/

SVN

pip install svn+svn://svn.myproject.org/svn/MyProject

Bzr

pip install bzr+http://bzr.myproject.org/MyProject/trunk

The following protocols are supported: [+git, +svn, +hg, +bzr]

Versions

@tag lets you specify a specific version/tag to check out.

#egg=name lets you specify what the project is as a dependency for others.

The order must always be @tag#egg=name.

Private Repositories

You can also install from private repositories by changing the protocol to SSH (ssh://) and adding an appropriate user (git@):

git+ssh://git@github.com/username/my_private_repo

You can also install from private repositories with a username / password.

git+https://<username>:<password>@github.com/<user>/<repo>.git

Github provides the ability to create personal OAuth tokens which can be cycled

git+https://<oauth token>:x-oauth-basic@github.com/<user>/<repo>.git

requirements.txt

requirements.txt is used to specify project dependencies:

requirements.txt

package1
package2==1.0.2
package3>=0.0.4
git+https://github.com/username/repo.git

These are not installed automatically with the package and must be installed with the command pip -r requirements.txt.

Including requirements files

Requirements files can include other requirements files:

requirements-docs.txt

sphinx
-r requirements-dev.txt

requirements-dev.txt

some-dev-tool
-r requirements.txt

requirements.txt

package1
package2==1.0.2
package3>=0.0.4
git+https://github.com/username/repo.git

setup.py

Requirements files can install dependencies specified in setup.py with the following command:

-e .

setup.py can also install from repositories using the same syntax as above, but using the dependency_links value as mentioned in this answer.

References:

https://pip.pypa.io/en/latest/user_guide.html#installing-packages https://pip.pypa.io/en/latest/reference/pip_install.html

Community
  • 1
  • 1
Rebs
  • 3,638
  • 1
  • 26
  • 28
  • 2
    setup.py CAN install from repositories. Just search of 'setup.py dependency_links' – TomDotTom Apr 29 '16 at 19:24
  • 1
    @TomDotTom Derp, I even upvoted that answer but somehow didn't assimilate it =P I'll update my answer. Thanks for pointing that out! It'll help out with some things I'm doing. – Rebs May 02 '16 at 03:18
  • dependency_links I think is being depreciated though (?) https://github.com/pypa/pip/issues/3939. I love this answer and think it is better due to being able (in setup.py): `extras_require={'all': [repo @ git+https://github.com/username/repo.git]}` – Josiah L. Nov 23 '19 at 18:25
23

As I just had to do the same thing, I found another way to do this as pip's --process-dependency-links are scheduled to be removed in pip 19.0 according to this comment.

pip 18.1 includes the following feature

Allow PEP 508 URL requirements to be used as dependencies.

From the description of PEP 508, the syntax for such URL dependencies looks like:

A minimal URL based lookup:

pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686

So in your setup.py it would look like

setup(
   ...
   install_requires = [
   ...
   'python-gearman @ https://github.com/mtai/python-gearman/archive/master.zip'
   ...
   ]
)

Notice, the link is an archive file and could also be a specific release or branch of a repository as described in this answer. Also, see that answer for working with other repository hosts.

To the best of my knowledge, the easiest way to update the dependency is by using pip install -I . when installing your package from its directory.

Community
  • 1
  • 1
Phil
  • 349
  • 2
  • 4
  • 1
    Does this also support everything that `pip install` supports, such as git URLs, `#subdirectory=...` etc? Or did they come up with a brand new syntax with different features exposed in a different and incompatible way? – remram Feb 12 '19 at 18:00
  • If you don't use setuptools and manage dependencies manually on the command line it seems like you still have to use the approach described by [@Rebs' answer](https://stackoverflow.com/a/23865528/10632788) . – Phil Feb 16 '19 at 09:27
  • 1
    Just to confirm this works with `distutils.core.setup` and `pip 19.1.1 ` – shouldsee May 15 '19 at 13:32
  • The resultant `requirements.txt`, however, is not compatible with `pip install -r requirments.txt` – shouldsee May 15 '19 at 14:20
  • @shouldsee which `requirments.txt` are you referring to? – Phil Jun 17 '19 at 13:55
  • I meant including line `python-gearman @ https://github.com/mtai/python-gearman/archive/master.zip` in `requirement.txt` would cause `pip install -r requirement.txt` to break. – shouldsee Jun 18 '19 at 12:08
  • Ahh, yes, it did not seem possible to use the same syntax in a `requirements.txt` when I discovered this, which is why my answer suggests using a `setup.py` file. – Phil Jun 18 '19 at 12:19
6

Vanilla setuptools does not support downloading directly from a git repository but you can use one of the Download Source links from that page, like:

easy_install http://github.com/mtai/python-gearman/tarball/master
ThiefMaster
  • 285,213
  • 77
  • 557
  • 610
Ned Deily
  • 78,314
  • 15
  • 120
  • 148
  • So, in order to make sure that this version of python-gearman is installed on any server where my package is going to be, I'm going to have to run easy_install manually before installing my package? – andrei Aug 12 '10 at 22:31
  • If you use easy_install, yes. But, as others have pointed out, you could switch to `pip` or `buildout` which have more sophisticated requirements management. See, for example: http://pip.openplans.org/#requirements-files – Ned Deily Aug 12 '10 at 22:52
  • Actually, you don't have to manually run easy_install; you can simply add the extra link to your setup.py. I'll write an answer explaining the details. – PJ Eby Aug 14 '10 at 00:05
  • 2
    As mentioned in my comment above setup.py provides dependency_links which allows you to download from a gti repository – TomDotTom Apr 29 '16 at 19:25