Quick intro to Python virtualenv on CS servers.
What is virtualenv & why should I care?
You want to run Numpy 1.13.3 but the rest of your group is still on 1.11.0. You opened a support ticket but after asking to install the 256th dependency you haven't heard back. We've all been there and this is where virtualenv comes in. It allows you to set up your own little Python garden that has all the packages and all the versions that you need. It even allows you to setup multiple gardens so you can even run python 2.7 with Tensoflow 1.2 and python 3.4 with Tensorflow 1.4. And you can do all of this without sudo or root privileges.
Where do I start?
First you need a good place to store your virtualenv. Here are some options:
- AFS home (~/): probably not the best idea as it has limited space and you'll run out if you need a lot of packages.
- Local scratch space (/scr, /lfs, *): good if you're running your experiments on a single server.
- Group shared space (/svgl, /dfs, **): good if you're running your experiments on a cluster or multiple servers and you want your virtualenv to be available on all of them.
Once you've decided on the best directory to store your virtualenv cd into it and run (we'll use /scr/mycsid in our examples and we're calling our virtualenv mypy3):
# Change into the directory where we'll store the virtualenv
cd /scr/mycsid
# Create a new virtualenv that will use python3 virtualenv -p /usr/bin/python3 mypy3
How do I use it?
You just activate it:
# Change into the directory where we'll store the virtualenv
cd /scr/mycsid/mypy3
# Activate
source bin/activate
You'll notice the name of the virtualenv (mypy3) displayed in parentheses in the prompt if you have successfully opened your virtualenv.
How do I exit it?
deactivate
Upgrade pip
Upgrading pip to the latest version is going to make your life a lot easier:
pip install --upgrade pip
Install different packages inside a virtualenv
Here are a few examples on how to get popular packages up and running inside your virtualenv. We are assuming you have activated your vitualenv and are currently in the /scr/mycsid/mypy3 directory.
Numpy
# Install pip install numpy # Test (p3)$ python >>> import numpy >>> numpy.__version__ '1.13.3'
OpenCV 3.3.0
Install numpy first.
# Download the repository
git clone https://github.com/Itseez/opencv.git
# Checkout the 3.3.0 version
cd opencv
git checkout 3.3.0
# Configure the build, use the path to your virtualenv
# for CMAKE_INSTALL_PREFIX (see above)
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/scr/mycisd/mypy3 \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=OFF \
-D BUILD_EXAMPLES=ON ..
# Compile, replace 4 with number of physical CPUs on your server
make -j4
# Install into your virtualenv
make install
# Test
python
>>> import cv2
>>> cv2.__version__
Tensorflow with GPU support
pip install --upgrade tensorflow-gpu
Keras
pip install keras
What if I am missing a dependancy?
Please submit a ticket at https://support.cs.stanford.edu.
References
* Check with your group where you can find the local scratch space. Most groups use the same naming conventions across all their servers.
** Check with your group where you can find the network mounted shared space. Most groups have one or more file servers that are shared with all the group's servers.
Comments