Differences between distribute, distutils, setuptools and distutils2?

Question

The Situation

I’m trying to port an open-source library to Python 3. (SymPy, if anyone is wondering.)

So, I need to run 2to3 automatically when building for Python 3. To do that, I need to use distribute. Therefore, I need to port the current system, which (according to the doctest) is distutils.

The Problem

Unfortunately, I’m not sure what’s the difference between these modules—distutils, distribute, setuptools. The documentation is sketchy as best, as they all seem to be a fork of one another, intended to be compatible in most circumstances (but actually, not all)…and so on, and so forth.

The Question

Could someone explain the differences? What am I supposed to use? What is the most modern solution? (As an aside, I’d also appreciate some guide on porting to Distribute, but that’s a tad beyond the scope of the question…)

How confusing? I am come to python from a Java/C++ background. In those situations, distribution is very straight forward. With python, I a, completely confused regarding all these distribution systems. — Raffi Khatchadourian, Nov 27 '11 at 06:33
I agree, Python packaging/installation has way too many alternatives with no clear guidance from the community. — Sabuncu, May 02 '12 at 18:19
I just wanted to link this related info on pip not supporting binary distributions http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere — pixelbeat, Jun 29 '12 at 15:54
@pixelbeat pip does support installing wheels (so-called binary distributions), that link is out-of-date. — Flimm, Mar 02 '20 at 15:11

Flimm · Accepted Answer · 2020-03-02T15:07:11.220

As of March 2020, most of the other answers to this question are several years out-of-date. When you come across advice on Python packaging issues, remember to look at the date of publication, and don't trust out-of-date information.

The Python Packaging User Guide is worth a read. Every page has a "last updated" date displayed, so you can check the recency of the manual, and it's quite comprehensive. The fact that it's hosted on a subdomain of python.org of the Python Software Foundation just adds credence to it. The Project Summaries page is especially relevant here.

Summary of tools:

Here's a summary of the Python packaging landscape:

Supported tools:

distutils is still the standard tool for packaging in Python. It is included in the standard library (Python 2 and Python 3). It is useful for simple Python distributions, but lacks features. It introduces the distutils Python package that can be imported in your setup.py script.
- _{Official docs | distutils section of Python Package User Guide}
setuptools was developed to overcome Distutils' limitations, and is not included in the standard library. It introduced a command-line utility called easy_install. It also introduced the setuptools Python package that can be imported in your setup.py script, and the pkg_resources Python package that can be imported in your code to locate data files installed with a distribution. One of its gotchas is that it monkey-patches the distutils Python package. It should work well with pip. It sees regular releases.
- _{Official docs | Pypi page | GitHub repo | setuptools section of Python Package User Guide}
scikit-build is an improved build system generator that internally uses CMake to build compiled Python extensions. Because scikit-build isn't based on distutils, it doesn't really have any of its limitations. When ninja-build is present, scikit-build can compile large projects over three times faster than the alternatives. It should work well with pip.
- _{Official docs | Pypi page | GitHub repo | scikit-build section of Python Package User Guide}
distlib is a library that provides functionality that is used by higher level tools like pip.
- _{Official Docs | Pypi page | Bitbucket repo | distlib section of Python Package User Guide}
packaging is also a library that provides functionality used by higher level tools like pip and setuptools
- _{Official Docs | Pypi page | GitHub repo | packaging section of Python Package User Guide}

Deprecated/abandoned tools:

distribute was a fork of setuptools. It shared the same namespace, so if you had Distribute installed, import setuptools would actually import the package distributed with Distribute. Distribute was merged back into Setuptools 0.7, so you don't need to use Distribute any more. In fact, the version on Pypi is just a compatibility layer that installs Setuptools.
distutils2 was an attempt to take the best of distutils, setuptools and distribute and become the standard tool included in Python's standard library. The idea was that distutils2 would be distributed for old Python versions, and that distutils2 would be renamed to packaging for Python 3.3, which would include it in its standard library. These plans did not go as intended, however, and currently, distutils2 is an abandoned project. The latest release was in March 2012, and its Pypi home page has finally been updated to reflect its death.

Others:

There are other tools, if you are interested, read Project Summaries in the Python Packaging User Guide. I won't list them all, to not repeat that page, and to keep the answer matching the question, which was only about distribute, distutils, setuptools and distutils2.

Recommendation:

If all of this is new to you, and you don't know where to start, I would recommend learning setuptools, along with pip and virtualenv, which all work very well together.

If you're looking into virtualenv, you might be interested in this question: What is the difference between venv, pyvenv, pyenv, virtualenv, virtualenvwrapper, etc?. (Yes, I know, I groan with you.)

Can anyone shed some light on what happened to distutils2? did the team move back to working on distribute? — vinc3m1, Mar 03 '13 at 19:35
@makeramen: See [this thread on the mailing list](http://mail.python.org/pipermail/python-dev/2012-June/120430.html). — Flimm, Mar 04 '13 at 21:43
And is not looking any better: `'Distribute' is a now deprecated fork of the 'Setuptools' project.` @ PyPI Distribute page. — KurzedMetal, May 29 '13 at 17:26
@KurzedMetal: I consider that deprecation warning on Distribute Pypi page a [bug](https://bitbucket.org/pypa/setuptools/issue/13/pypi-home-pages-of-setuptools-and), as Setuptools Pypi page has not been updated. — Flimm, Jun 04 '13 at 08:00
@KurzedMetal, according to the SetupTools folks, setuptools 0.7 will subsume both distribute and the old setuptools restoring order to the universe. So things actually are set to improve considerably! — John McDonnell, Jun 05 '13 at 04:27
The [Python Packaging User Guide](https://python-packaging-user-guide.readthedocs.org/en/latest/index.html) will have the most up-to-date info on state of python packaging. It was noted by Nick Coughlan at the [2013 PyCon](https://www.youtube.com/watch?v=ePFWp3oSfyU). — imanuelcostigan, Jul 06 '13 at 08:11
@Amro: `pip` is not an alternative to these tools but it does help with installing packages. — Flimm, Jul 11 '13 at 08:48
@Flimm: absolutely, but I feel like it deserves more than a short sentence under setuptools (which it depends upon) :) Anyway thanks for the comprehensive answer — Amro, Jul 11 '13 at 12:56
If you're going to mention Bento, you should also mention Conda (http://conda.io) which is the core open-source technology used in the popular Anaconda distribution. — Peter Wang, Apr 21 '15 at 22:31
@Flimm The last edit is quite a while ago. Is Bento mature by now? — Martin Thoma, Jan 03 '16 at 09:01
The link to the docs of `distlib` is outdated - newest is https://distlib.readthedocs.io/en/latest/index.html. Their self-described project status has also been upgraded from alpha: "The project has reached a mature status in its development: there is a test suite and it has been exercised on Windows, Ubuntu and Mac OS X. The project is used by well-known projects such as pip and caniusepython3." — Axel, Feb 07 '18 at 15:11
Does Setuptools work very well with Anaconda (as opposed to Virtualenv and Pip)? — Luis B, Mar 11 '19 at 19:44
Ironically, this answer is now also badly out of date. [PEP 517](https://www.python.org/dev/peps/pep-0517/), poetry, flit, `pyproject.toml` ... — wim, Oct 25 '19 at 06:43
I've updated the answer post. I didn't talk about Bento, Anaconda, wheels, and so on, to stop the answer from getting too big and from drifting too far from the original question. — Flimm, Mar 02 '20 at 15:07
You are a god, thanks for keep this maintained, I have this bookmarked and from time to time I comeback to see if I missed any changes, I've seen quite a bunch of the updates of this answer. Again: thank you very much for you time, like you said there's a lot of misinformation around, and I'm glad to have this as a reliable source of updated info. — KurzedMetal, Jul 03 '20 at 18:15

Éric Araujo · Answer 2 · 2013-08-13T23:17:16.360

252

I’m a distutils maintainer and distutils2/packaging contributor. I did a talk about Python packaging at ConFoo 2011 and these days I’m writing an extended version of it. It’s not published yet, so here are excerpts that should help define things.

Distutils is the standard tool used for packaging. It works rather well for simple needs, but is limited and not trivial to extend.
Setuptools is a project born from the desire to fill missing distutils functionality and explore new directions. In some subcommunities, it’s a de facto standard. It uses monkey-patching and magic that is frowned upon by Python core developers.
Distribute is a fork of Setuptools that was started by developers feeling that its development pace was too slow and that it was not possible to evolve it. Its development was considerably slowed when distutils2 was started by the same group. 2013-August update: distribute is merged back into setuptools and discontinued.
Distutils2 is a new distutils library, started as a fork of the distutils codebase, with good ideas taken from setup tools (of which some were thoroughly discussed in PEPs), and a basic installer inspired by pip. ~~The actual name you use to import Distutils2 is packaging in the Python 3.3+ standard library, or distutils2 in 2.4+ and 3.1–3.2. (A backport will be available soon.)~~ Distutils2 did not make the Python 3.3 release, and it was put on hold.

More info:

I hope to finish my guide soon, it will contain more info about each library’s strong and weak points and a transition guide.

edited Aug 13 '13 at 23:17

answered Jun 29 '11 at 15:07

Éric Araujo

6,406
1
23
37

@Éric Araujo Please help me understand this, will distutils2 and distribute one day converge in a unique tool/library? Thanks. – Paolo Jul 11 '11 at 13:32
1

No. distutils2 takes some good ideas from setuptools/distribute, after standardization (PEPs) or not (for example, I mentor a GSoC student who’s adding a develop command and automatic scripts generation), but it won’t ever be a drop-in replacement: there are some parts we don’t want (eggs, VCS integration, etc.). OTOH, distutils2 has some things that setuptools/distribute have not. To ease transition, I think the distribute developers maybe will use distutils2 to support new standards and tools; I also think I remember the setuptools developer saying that he wants to support new standards. – Éric Araujo Jul 29 '11 at 14:09
1

Where does ez_setup fall in all this? Also are there any updates to the status of distutils2? – James McMahon Jun 24 '12 at 15:56
ez_setup is a bootstrap script that people can include in their repositories and source distributions so that setuptools is automatically downloaded and used when running setup.py. distribute_setup is its distribute equivalent. The maintainer of pip and I think that it’s inelegant and people should be instructed to install setuptools before running the setup script instead. – Éric Araujo Jun 25 '12 at 06:14
1

distutils2 is still slowly progressing. Its inclusion in the standard library has been deferred from 3.3 to 3.4 because it was not ready. For most people that does not make a difference as they use 2.x and can get d2 with pip. – Éric Araujo Jun 25 '12 at 06:15
2

@ÉricAraujo Sorry to hear about the delay. I really hope it’s ready in time for 3.4! I **love** Python, but the packaging has always made me bang my head against the wall. (In other news, how is your guide coming? If it’s finished, could you link it in your answer above?) – Zearin Oct 14 '12 at 22:03
9

@AlexisHuet This kind of comment would be better if it would contain the link to the [comment below](http://stackoverflow.com/a/14753678/131120) (which you can get from the `share` button). – erikbwork Mar 26 '13 at 15:26
2

you should perhaps update the answer to mention that `distribute` was recently merged back in `setuptools`. The fact that much of the information out-there is out-dated adds to the confusion – Amro Jul 10 '13 at 22:26
from the post here: http://stackoverflow.com/questions/6587507/how-to-install-pip-with-python-3 "distribute has since been superseded by [setup_tools] (pypi.python.org/pypi/setuptools). – @wegry Aug 3 at 22:07" I'm assuming this is a false statement based on your post. – franklin Aug 10 '13 at 15:39
Distribute was merged back into setuptools, so the latest release of setuptools does supersede distribute. – Éric Araujo Aug 13 '13 at 23:16
So is the best current choice distutils2, or not? Setuptools bugs me downloading .eggs to the current project directory. Distutils2 seems to let you specify a path. Can't work out what the current state of all these tools is though. So confusing to a newcomer. Other languages just ship one system and stick with it. The Python community seems like it just wants to keep swapping and changing things. – d11wtq Jun 14 '14 at 10:53

Keith · Answer 3 · 2019-10-27T01:07:02.597

5

NOTE: Answer deprecated, Distribute now obsolete. This answer is no longer valid since the Python Packaging Authority was formed and has done a lot of work cleaning this up.

Yep, you got it. :-o I think at this time the preferred package is Distribute, which is a fork of setuptools, which are an extension of distutils (the original packaging system). Setuptools was not being maintained so is was forked and renamed, however when installed it uses the package name of setuptools! I think most Python developers now use Distribute, and I can say for sure that I do.

edited Oct 27 '19 at 01:07

answered Jun 14 '11 at 14:00

Keith

37,985
10
48
67

For the record, I accepted this answer because it told me the situation now (And the is fork of is extension of relation that the picture in the other answer just doesn't mention). And somewhere along the road I also learned that the documentation itself isn't usually sure what it's trying to say. – VPeric Jun 23 '11 at 12:26
2

@VPeric, Indeed, the documentation reflects the fact that this aspect of python is in a state of flux/ a mess. – juanchopanza Jun 23 '11 at 14:19

score 2 · Answer 4 · answered Sep 29 '14 at 21:26

Updating this question in late 2014 where fortunately the Python packaging chaos has been greatly cleaned up by Continuum's "conda" package manager.

In particular, conda quickly enables the creation of conda "environments". You can configure your environments with different versions of Python. For example:

conda create -n py34 python=3.4 anaconda

conda create -n py26 python=2.6 anaconda

will create two ("py34" or "py26") Python environments with different versions of Python.

Afterwards you can invoke the environment with the specific version of Python with:

source activate <env name>

This feature seems especially useful in your case where you are having to deal with different version of Python.

Moreover, conda has the following features:

Python agnostic
Cross platform
No admin privileges required
Smart dependency management (by way of a SAT solver)
Nicely deals with C, Fortran and system level libraries that you may have to link against

That last point is especially important if you are in the scientific computing arena.

score 2 · Answer 5 · answered Jul 29 '11 at 14:16

I realize that I have replied to your secondary question without addressing unquestioned assumptions in your original problem:

I'm trying to port an open-source library (SymPy, if anyone is wondering) to Python 3. To do this, I need to run 2to3 automatically when building for Python 3.

You may, not need. Other strategies are described at http://docs.python.org/dev/howto/pyporting

To do that, I need to use distribute,

You may :) distutils supports build-time 2to3 conversion for code (not docstrings), in a different manner that distribute’s: http://docs.python.org/dev/howto/pyporting#during-installation

Thanks, though we've already decided to solve the problem by writing our script to handle the conversion. And yeah, I knew there were other options than using 2to3, but SymPy is a complex codebase (around 200k+ lines last time I checked) and using 2to3 was the only realistic strategy. Thanks again, in any case! — VPeric, Jul 31 '11 at 20:49