How to install packages offline?

Question

What's the best way to download a python package and it's dependencies from pypi for offline installation on another machine? Is there any easy way to do this with pip or easy_install? I'm trying to install the requests library on a FreeBSD box that is not connected to the internet.

Answer 1

The pip download command lets you download packages without installing them:

pip download -r requirements.txt

(In previous versions of pip, this was spelled pip install --download -r requirements.txt.)

Then you can use pip install --no-index --find-links /path/to/download/dir/ -r requirements.txt to install those downloaded sdists, without accessing the network.

Answer 2

If you want install python libs and their dependencies offline, finish following these steps on a machine with the same os, network connected, and python installed:

1) Create a requirements.txt file with similar content (Note - these are the libraries you wish to download):

Flask==0.12
requests>=2.7.0
scikit-learn==0.19.1
numpy==1.14.3
pandas==0.22.0

One option for creating the requirements file is to use pip freeze > requirements.txt. This will list all libraries in your environment. Then you can go in to requirements.txt and remove un-needed ones.

2) Execute command mkdir wheelhouse && pip download -r requirements.txt -d wheelhouse to download libs and their dependencies to directory wheelhouse

3) Copy requirements.txt into wheelhouse directory

4) Archive wheelhouse into wheelhouse.tar.gz with tar -zcf wheelhouse.tar.gz wheelhouse

Then upload wheelhouse.tar.gz to your target machine:

1) Execute tar -zxf wheelhouse.tar.gz to extract the files

2) Execute pip install -r wheelhouse/requirements.txt --no-index --find-links wheelhouse to install the libs and their dependencies

Answer 3

If the package is on PYPI, download it and its dependencies to some local directory. E.g.

$ mkdir /pypi && cd /pypi
$ ls -la
  -rw-r--r--   1 pavel  staff   237954 Apr 19 11:31 Flask-WTF-0.6.tar.gz
  -rw-r--r--   1 pavel  staff   389741 Feb 22 17:10 Jinja2-2.6.tar.gz
  -rw-r--r--   1 pavel  staff    70305 Apr 11 00:28 MySQL-python-1.2.3.tar.gz
  -rw-r--r--   1 pavel  staff  2597214 Apr 10 18:26 SQLAlchemy-0.7.6.tar.gz
  -rw-r--r--   1 pavel  staff  1108056 Feb 22 17:10 Werkzeug-0.8.2.tar.gz
  -rw-r--r--   1 pavel  staff   488207 Apr 10 18:26 boto-2.3.0.tar.gz
  -rw-r--r--   1 pavel  staff   490192 Apr 16 12:00 flask-0.9-dev-2a6c80a.tar.gz

Some packages may have to be archived into similar looking tarballs by hand. I do it a lot when I want a more recent (less stable) version of something. Some packages aren't on PYPI, so same applies to them.

Suppose you have a properly formed Python application in ~/src/myapp. ~/src/myapp/setup.py will have install_requires list that mentions one or more things that you have in your /pypi directory. Like so:

  install_requires=[
    'boto',
    'Flask',
    'Werkzeug',
    # and so on

If you want to be able to run your app with all the necessary dependencies while still hacking on it, you'll do something like this:

$ cd ~/src/myapp
$ python setup.py develop --always-unzip --allow-hosts=None --find-links=/pypi

This way your app will be executed straight from your source directory. You can hack on things, and then rerun the app without rebuilding anything.

If you want to install your app and its dependencies into the current python environment, you'll do something like this:

$ cd ~/src/myapp
$ easy_install --always-unzip --allow-hosts=None --find-links=/pypi .

In both cases, the build will fail if one or more dependencies aren't present in /pypi directory. It won't attempt to promiscuously install missing things from Internet.

I highly recommend to invoke setup.py develop ... and easy_install ... within an active virtual environment to avoid contaminating your global Python environment. It is (virtualenv that is) pretty much the way to go. Never install anything into global Python environment.

If the machine that you've built your app has same architecture as the machine on which you want to deploy it, you can simply tarball the entire virtual environment directory into which you easy_install-ed everything. Just before tarballing though, you must make the virtual environment directory relocatable (see --relocatable option). NOTE: the destination machine needs to have the same version of Python installed, and also any C-based dependencies your app may have must be preinstalled there too (e.g. say if you depend on PIL, then libpng, libjpeg, etc must be preinstalled).

Answer 4

offline python. for doing this I use virtualenv (isolated Python environment)

1) install virtualenv online with pip:

pip install virtualenv --user

or offline with whl: go to this link , download last version (.whl or tar.gz) and install that with this command:

pip install virtualenv-15.1.0-py2.py3-none-any.whl --user

by using --user you don't need to use sudo pip….

2) use virtualenv

on online machine select a directory with terminal cd and run this code:

python -m virtualenv myenv
cd myenv
source bin/activate
pip install Flask

after installing all the packages, you have to generate a requirements.txt so while your virtualenv is active, write

pip freeze > requirements.txt

open a new terminal and create another env like myenv2.

python -m virtualenv myenv2
cd myenv2
source bin/activate
cd -
ls

now you can go to your offline folder where your requirements.txt and tranferred_packages folder are in there. download the packages with following code and put all of them to tranferred_packages folder.

pip download -r requirements.txt

take your offline folder to offline computer and then

python -m virtualenv myenv2
cd myenv2
source bin/activate
cd -
cd offline
pip install --no-index --find-links="./tranferred_packages" -r requirements.txt

what is in the folder offline [requirements.txt , tranferred_packages {Flask-0.10.1.tar.gz, ...}]

check list of your package

pip list

note: as we are in 2017 it is better to use python 3. you can create python 3 virtualenv with this command.

virtualenv -p python3 envname

Answer 5

Let me go through the process step by step:

1. On a computer connected to the internet, create a folder.

   $ mkdir packages
   $ cd packages

2. open up a command prompt or shell and execute the following command:

   Suppose the package you want is tensorflow

   $ pip download tensorflow

3. Now, on the target computer, copy the packages folder and apply the following command

  $ cd packages
  $ pip install 'tensorflow-xyz.whl' --no-index --find-links '.'

Note that the tensorflow-xyz.whl must be replaced by the original name of the required package.

Answer 6

I had a similar problem. And i had to make it install the same way, we do from pypi.

I did the following things:

1. Make a directory to store all the packages in the machine that have internet access.

   mkdir -p /path/to/packages/
   

2. Download all the packages to the path

Edit: You can also try:

python3 -m pip wheel --no-cache-dir -r requirements.txt -w /path/to/packages

pip download -r requirements.txt -d /path/to/packages

Eg:- ls /root/wheelhouse/  # **/root/wheelhouse** is my **/path/to/packages/**
total 4524
-rw-r--r--. 1 root root   16667 May 23  2017 incremental-17.5.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root   34713 Sep  1 10:21 attrs-18.2.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root 3088398 Oct 15 14:41 Twisted-18.9.0.tar.bz2
-rw-r--r--. 1 root root  133356 Jan 28 15:58 chardet-3.0.4-py2.py3-none-any.whl
-rw-r--r--. 1 root root  154154 Jan 28 15:58 certifi-2018.11.29-py2.py3-none-any.whl
-rw-r--r--. 1 root root   57987 Jan 28 15:58 requests-2.21.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root   58594 Jan 28 15:58 idna-2.8-py2.py3-none-any.whl
-rw-r--r--. 1 root root  118086 Jan 28 15:59 urllib3-1.24.1-py2.py3-none-any.whl
-rw-r--r--. 1 root root   47229 Jan 28 15:59 tqdm-4.30.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root    7922 Jan 28 16:13 constantly-15.1.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root  164706 Jan 28 16:14 zope.interface-4.6.0-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r--. 1 root root  573841 Jan 28 16:14 setuptools-40.7.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root   37638 Jan 28 16:15 Automat-0.7.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root   37905 Jan 28 16:15 hyperlink-18.0.0-py2.py3-none-any.whl
-rw-r--r--. 1 root root   52311 Jan 28 16:15 PyHamcrest-1.9.0-py2.py3-none-any.whl
 -rw-r--r--. 1 root root   10586 Jan 28 16:15 six-1.12.0-py2.py3-none-any.whl

3. Tar the packages directory and copy it to the Machine that doesn't have internet access. Then do,

   cd /path/to/packages/
   tar -cvzf packages.tar.gz .  # not the . (dot) at the end
   

Copy the packages.tar.gz into the destination machine that doesn't have internet access.

4. In the machine that doesn't have internet access, do the following (Assuming you copied the tarred packages to /path/to/package/ in the current machine)

   cd /path/to/packages/
   tar -xvzf packages.tar.gz
   mkdir -p $HOME/.config/pip/
   vi $HOME/.config/pip/pip.conf
   

and paste the following content inside and save it.

[global]
timeout = 10
find-links = file:///path/to/package/
no-cache-dir = true
no-index = true

5. Finally, i suggest you use, some form of virtualenv for installing the packages.

   virtualenv -p python2 venv # use python3, if you are on python3
   source ./venv/bin/activate
   pip install <package>
   

You should be able to download all the modules that are in the directory /path/to/package/.

Note: I only did this, because i couldn't add options or change the way we install the modules. Otherwise i'd have done

pip install --no-index --find-links /path/to/download/dir/ -r requirements.txt

Answer 7

Download the tarball, transfer it to your FreeBSD machine and extract it, afterwards run python setup.py install and you're done!

EDIT: Just to add on that, you can also install the tarballs with pip now.

Answer 8

Using wheel compiled packages.

bundle up:

$ tempdir=$(mktemp -d /tmp/wheelhouse-XXXXX)
$ pip wheel -r requirements.txt --wheel-dir=$tempdir
$ cwd=`pwd`
$ (cd "$tempdir"; tar -cjvf "$cwd/bundled.tar.bz2" *)

copy tarball and install:

$ tempdir=$(mktemp -d /tmp/wheelhouse-XXXXX)
$ (cd $tempdir; tar -xvf /path/to/bundled.tar.bz2)
$ pip install --force-reinstall --ignore-installed --upgrade --no-index --no-deps $tempdir/*

Note wheel binary packages are not across machines.

More ref. here: https://pip.pypa.io/en/stable/user_guide/#installation-bundles

Answer 9

For Pip 8.1.2 you can use pip download -r requ.txt to download packages to your local machine.

Answer 10

Download the wheel file (for example dlb-0.5.0-py3-none-any.whl) from Pypi and

pip install dlb-0.5.0-py3-none-any.whl

Answer 11

As a continues to @chaokunyang answer, I want to put here the script I write that does the work:

1. Write a "requirements.txt" file that specifies the libraries you want to pack.
2. Create a tar file that contains all your libraries (see the Packer script).
3. Put the created tar file in the target machine and untar it.
4. run the Installer script (which is also packed into the tar file).

"requirements.txt" file

docker==4.4.0

Packer side: file name: "create-offline-python3.6-dependencies-repository.sh"

#!/usr/bin/env bash

# This script follows the steps described in this link:
# https://stackoverflow.com/a/51646354/8808983

LIBRARIES_DIR="python3.6-wheelhouse"

if [ -d ${LIBRARIES_DIR} ]; then
    rm -rf ${LIBRARIES_DIR}/*
else
    mkdir ${LIBRARIES_DIR}
fi

pip download -r requirements.txt -d ${LIBRARIES_DIR}

files_to_add=("requirements.txt" "install-python-libraries-offline.sh")

for file in "${files_to_add[@]}"; do
    echo "Adding file ${file}"
    cp "$file" ${LIBRARIES_DIR}
done

tar -cf ${LIBRARIES_DIR}.tar ${LIBRARIES_DIR}

Installer side: file name: "install-python-libraries-offline.sh"

#!/usr/bin/env bash

# This script follows the steps described in this link:
# https://stackoverflow.com/a/51646354/8808983

# This file should run during the installation process from inside the libraries directory, after it was untared:
# pythonX-wheelhouse.tar -> untar -> pythonX-wheelhouse
# |
# |--requirements.txt
# |--install-python-libraries-offline.sh


pip3 install -r requirements.txt --no-index --find-links .