Pipenv

Edit 2021:

If you like pipenv’s workflow you can also look into poetry.

Handling dependencies

There are a couple of way to organize a Python project, in terms of handling dependencies. The reason why this is important, is that you want control over the environment, and exactly what package versions your project is dependent upon. Thus, it is highly recommended that you use one Python environment per project.

From a scientific point of view this is very important for reproducibility.

The most common ways of handling dependencies for a small project would be to use Pipenv or Conda. On a larger scale Docker is popularly used since it is actually a virtual environment for a whole system (container) and not just for Python, thus it can handle dependencies on a larger scale.

However, all this can become messy.

Besides from Pipenv and Docker, there are many other tools, e.g., Venv and Virtualenv.

Pip

Pip is a package manager for Python and is the recommended way to install packages. E.g.,

sudo apt-get install python3-pip

Once you have installed pip you can check that it works by,

python3
import pip

Now you can either run pip from the CLI using Python’s module argument,

python -m pip <command> [options]

or by using the pip3 command, which should be located under /usr/bin/pip3,

pip3 <command> [options]

Pipenv

Pipenv, with its slogan “Python Dev Workflow for Humans”, is a tool to manage Python package for a specific project. It brings together Virtualenv and Pip, to simplify working with dependencies. This means that with Pipenv you don’t have to use Pip together with Virtualenv. Pipenv also handles what version of Python that the project is running, and everything is located in an virtual environment under the Pipenv directory. To make a user install (recommended),

pip3 install --user pipenv

Note that this program is located at /home/magnus/.local/bin/pipenv.

Now you can cd into your project directory and install packages with,

pipenv install <packages>

This will both install the package and add it as a dependency in the Pipfile.

There is a nice tool where you can try out Pipenv online, called, Pipenv playground.

Note that, Pipenv supersedes requirements.txt and instead uses Pipfiles for the dependencies of the project. If, e.g., you clone a repo with a Pipfile in, you simply install Pipenv, and run pipenv install.

Structuring your Python project

Setting up a good directory structure is important for a project. A good guide can be found here.

How to use Pipenv

Once Pipenv is installed you don’t have to care about Pip anymore. Pipfile replaces requirements.txt and Pipfile.lock ensures a deterministic build of your project. Pipenv uses pip and virtualenv in the background, but join their functions to one application.

Using Pipenv,

cd to your project folder and spawn a shell within the virtualenv,

pipenv shell

This will create a virtual environment if one doesn’t already exist,

Creating a virtualenv for this project…
Pipfile: /home/magnus/your_project/Pipfile
Using /usr/bin/python3 (3.7.3) to create virtualenv…
...
Launching subshell in virtual environment…

At this point a subshell has been opened. Here you can install packages, either with or without specifying a specific version,

pipenv install numpy

Numpy will be installed, and a Pipfile.lock will be created. Note that this can take some time if you are building with large packages such as NumPy. When this is done you can run,

pipenv graph

outputting the dependencies, in this case, numpy==1.18.2.

Once you are in the virtual shell, you can run python as usual executing your scripts. Either you can run python script.py or pipenv run python script.py. The first can only be run in the virtual shell, however the second I think could be run outside.

Here you can have an alias, e.g.,

prp='pipenv run python'

Running pipenv lock will update the Pipfile.lock, exit will exit the virtual shell. pipenv --venv will give you where the virtual environment is located, it’s a good idea to investigate the structure of it. Similarly pipenv --py will show which Python is in the environment.

NOTE: All these command needs to be run from the directory, thus you need to cd into where your environment is located. If you run, e.g., pipenv --where at your home directory, the following will be outputted,

No Pipfile present at project home. Consider running `pipenv install` first to automatically
generate a Pipfile for you.

Furthermore, you can use pipenv install --dev <package> to flag the package as a development package. In fact, Pipenv can track these two separate environments.