Getting started with virtualenv

Quick intro to Python virtualenv on CS servers.

What is virtualenv & why should I care?

You want to run Numpy 1.13.3 but the rest of your group is still on 1.11.0. You opened a support ticket but after asking to install the 256th dependency you haven't heard back. We've all been there and this is where virtualenv comes in. It allows you to set up your own little Python garden that has all the packages and all the versions that you need. It even allows you to setup multiple gardens so you can even run python 2.7 with Tensoflow 1.2 and python 3.4 with Tensorflow 1.4. And you can do all of this without sudo or root privileges.

Where do I start?

First you need a good place to store your virtualenv. Here are some options:

  1. AFS home (~/): probably not the best idea as it has limited space and you'll run out if you need a lot of packages.
  2. Local scratch space (/scr, /lfs, *): good if you're running your experiments on a single server.
  3. Group shared space (/svgl, /dfs, **): good if you're running your experiments on a cluster or multiple servers and you want your virtualenv to be available on all of them.

Once you've decided on the best directory to store your virtualenv cd into it and run (we'll use /scr/mycsid in our examples and we're calling our virtualenv mypy3):

# Change into the directory where we'll store the virtualenv
cd /scr/mycsid

# Create a new virtualenv that will use python3 virtualenv -p /usr/bin/python3 mypy3

How do I use it? 

You just activate it:

# Change into the directory where we'll store the virtualenv
cd /scr/mycsid/mypy3
# Activate 
source bin/activate

You'll notice the name of the virtualenv (mypy3) displayed in parentheses in the prompt if you have successfully opened your virtualenv.

How do I exit it?

deactivate

Upgrade pip

Upgrading pip to the latest version is going to make your life a lot easier:

pip install --upgrade pip

Install different packages inside a virtualenv

Here are a few examples on how to get popular packages up and running inside your virtualenv. We are assuming you have activated your vitualenv and are currently in the /scr/mycsid/mypy3 directory.

Numpy

# Install
pip install numpy

# Test
(p3)$ python
>>> import numpy
>>> numpy.__version__
'1.13.3'

OpenCV 3.3.0

Install numpy first.

# Download the repository
git clone https://github.com/Itseez/opencv.git

# Checkout the 3.3.0 version
cd opencv
git checkout 3.3.0

# Configure the build, use the path to your virtualenv
# for CMAKE_INSTALL_PREFIX (see above)
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
	-D CMAKE_INSTALL_PREFIX=/scr/mycisd/mypy3 \
	-D INSTALL_C_EXAMPLES=OFF \
	-D INSTALL_PYTHON_EXAMPLES=OFF \
	-D BUILD_EXAMPLES=ON ..

# Compile, replace 4 with number of physical CPUs on your server
make -j4

# Install into your virtualenv
make install

# Test
python
>>> import cv2
>>> cv2.__version__

Tensorflow with GPU support

pip install --upgrade tensorflow-gpu

Keras

pip install keras

What if I am missing a dependancy?

Please submit a ticket at https://support.cs.stanford.edu.

References

  1. Virtualenv User Guide, 2017.
  2. Install OpenCV 3.0 and Python 3.4+ on Ubuntu 14.04, 2015.

* Check with your group where you can find the local scratch space. Most groups use the same naming conventions across all their servers.

** Check with your group where you can find the network mounted shared space. Most groups have one or more file servers that are shared with all the group's servers. 

Have more questions? Submit a request

Comments