Is it bad to have my virtualenv directory inside my git repository?

Question

I'm thinking about putting the virtualenv for a Django web app I am making inside my git repository for the app. It seems like an easy way to keep deploy's simple and easy. Is there any reason why I shouldn't do this?

Answer 1

I use pip freeze to get the packages I need into a requirements.txt file and add that to my repository. I tried to think of a way of why you would want to store the entire virtualenv, but I could not.

Answer 2

Storing the virtualenv directory inside git will, as you noted, allow you to deploy the whole app by just doing a git clone (plus installing and configuring Apache/mod_wsgi). One potentially significant issue with this approach is that on Linux the full path gets hard-coded in the venv's activate, django-admin.py, easy_install, and pip scripts. This means your virtualenv won't entirely work if you want to use a different path, perhaps to run multiple virtual hosts on the same server. I think the website may actually work with the paths wrong in those files, but you would have problems the next time you tried to run pip.

The solution, already given, is to store enough information in git so that during the deploy you can create the virtualenv and do the necessary pip installs. Typically people run pip freeze to get the list then store it in a file named requirements.txt. It can be loaded with pip install -r requirements.txt. RyanBrady already showed how you can string the deploy statements in a single line:

# before 15.1.0
virtualenv --no-site-packages --distribute .env &&\
    source .env/bin/activate &&\
    pip install -r requirements.txt

# after deprecation of some arguments in 15.1.0
virtualenv .env && source .env/bin/activate && pip install -r requirements.txt

Personally, I just put these in a shell script that I run after doing the git clone or git pull.

Storing the virtualenv directory also makes it a bit trickier to handle pip upgrades, as you'll have to manually add/remove and commit the files resulting from the upgrade. With a requirements.txt file, you just change the appropriate lines in requirements.txt and re-run pip install -r requirements.txt. As already noted, this also reduces "commit spam".

Answer 3

I used to do the same until I started using libraries that are compiled differently depending on the environment such as PyCrypto. My PyCrypto mac wouldn't work on Cygwin wouldn't work on Ubuntu.

It becomes an utter nightmare to manage the repository.

Either way I found it easier to manage the pip freeze & a requirements file than having it all in git. It's cleaner too since you get to avoid the commit spam for thousands of files as those libraries get updated...

Answer 4

I think one of the main problems which occur is that the virtualenv might not be usable by other people. Reason is that it always uses absolute paths. So if you virtualenv was for example in /home/lyle/myenv/ it will assume the same for all other people using this repository (it must be exactly the same absolute path). You can't presume people using the same directory structure as you.

Better practice is that everybody is setting up their own environment (be it with or without virtualenv) and installing libraries there. That also makes you code more usable over different platforms (Linux/Windows/Mac), also because virtualenv is installed different in each of them.

Answer 5

It's not a good idea to include any environment-dependent component or setting in your repos as one of the key aspects of using a repo, is perhaps, sharing it with other developers. Here is how I would setup my development environment on a Windows PC (say, Win10).

1. Open Pycharm and on the first page, choose to check out the project from your Source Control System (in my case, I am using github)

2. In Pycharm, navigate to settings and choose "Project Interpreter" and choose the option to add a new virtual environment , you can call it "venv".

3. Choose the base python interpreter which is located at C:\Users{user}\AppData\Local\Programs\Python\Python36 (make sure you choose the appropriate version of Python based on what you have installed)

4. Note that Pycharm will create the new virtual environment and copy python binaries and required libraries under your venv folder inside your project folder.

5. Let Pycharm complete its scanning as it needs to rebuild/refresh your project skeleton

6. exclude venv folder from your git interactions (add venv\ to .gitignore file in your project folder)

Bonus: If you want people to easily (well, almost easily) install all the libraries your software needs, you can use

pip freeze > requirements.txt

and put the instruction on your git so people can use the following command to download all required libraries at once.

pip install -r requirements.txt 

Answer 6

I use what is basically David Sickmiller's answer with a little more automation. I create a (non-executable) file at the top level of my project named activate with the following contents:

[ -n "$BASH_SOURCE" ] \
    || { echo 1>&2 "source (.) this with Bash."; exit 2; }
(
    cd "$(dirname "$BASH_SOURCE")"
    [ -d .build/virtualenv ] || {
        virtualenv .build/virtualenv
        . .build/virtualenv/bin/activate
        pip install -r requirements.txt
    }
)
. "$(dirname "$BASH_SOURCE")/.build/virtualenv/bin/activate"

(As per David's answer, this assumes you're doing a pip freeze > requirements.txt to keep your list of requirements up to date.)

The above gives the general idea; the actual activate script (documentation) that I normally use is a bit more sophisticated, offering a -q (quiet) option, using python when python3 isn't available, etc.

This can then be sourced from any current working directory and will properly activate, first setting up the virtual environment if necessary. My top-level test script usually has code along these lines so that it can be run without the developer having to activate first:

cd "$(dirname "$0")"
[[ $VIRTUAL_ENV = $(pwd -P) ]] || . ./activate

Sourcing ./activate, not activate, is important here because the latter will find any other activate in your path before it will find the one in the current directory.

Answer 7

If you know which operating systems your application will be running on, I would create one virtualenv for each system and include it in my repository. Then I would make my application detect which system it is running on and use the corresponding virtualenv.

The system could e.g. be identified using the platform module.

In fact, this is what I do with an in-house application I have written, and to which I can quickly add a new system's virtualenv in case it is needed. This way, I do not have to rely on that pip will be able to successfully download the software my application requires. I will also not have to worry about compilation of e.g. psycopg2 which I use.

If you do not know which operating system your application may run on, you are probably better off using pip freeze as suggested in other answers here.

Answer 8

I think is that the best is to install the virtual environment in a path inside the repository folder, maybe is better inclusive to use a subdirectory dedicated to the environment (I have deleted accidentally my entire project when force installing a virtual environment in the repository root folder, good that I had the project saved in its latest version in Github).

Either the automated installer, or the documentation should indicate the virtualenv path as a relative path, this way you won't run into problems when sharing the project with other people. About the packages, the packages used should be saved by pip freeze -r requirements.txt.

Answer 9

If you just setting up development env, then use pip freeze file, caz that makes the git repo clean.

Then if doing production deployment, then checkin the whole venv folder. That will make your deployment more reproducible, not need those libxxx-dev packages, and avoid the internet issues.

So there are two repos. One for your main source code, which includes a requirements.txt. And a env repo, which contains the whole venv folder.