Python¶
Description¶
This page covers installing your own version of Python using the virtualenv command.
Your own version of Python is needed when your application or code requires additional Python packages, or a specific version of Python or Python packages. This gives you the control over what version of Python packages are installed and allows you to upgrade them if you wish so. System Python upgrades will not affect your Python virtual environment.
System Python¶
Default Python versions available on full infrastructure include:
python2 --version
Python 2.7.5
python3 --version
Python 3.6.8
Python2
We do not recommend anyone to use Python 2 anymore, since Python 2 is offically EOL. Only use python2 module if you have a legacy code that can't be upgrade to run on Python 3.
Furthermore, there are several different version of Python that are available as modules, and can be view by module avail command and loaded with module load:
module avail Python
---------------------------------------------- /storage-apps/easybuild/modules/lang ----------------------------------------------
Python/2.7.18-GCCcore-9.3.0 Python/3.8.6-GCCcore-10.2.0 Python/3.10.4-GCCcore-11.3.0-bare
Python/2.7.18-GCCcore-10.2.0 Python/3.9.5-GCCcore-10.3.0-bare Python/3.10.4-GCCcore-11.3.0
Python/2.7.18-GCCcore-11.2.0-bare Python/3.9.5-GCCcore-10.3.0 Python/3.10.8-GCCcore-12.2.0-bare
Python/2.7.18-GCCcore-11.3.0-bare Python/3.9.6-GCCcore-11.2.0-bare Python/3.10.8-GCCcore-12.2.0 (D)
Python/3.8.2-GCCcore-9.3.0 Python/3.9.6-GCCcore-11.2.0 SciPy-bundle/2020.03-foss-2020a-Python-3.8.2
---------------------------------------------- /storage-apps/easybuild/modules/lib -----------------------------------------------
pybind11/2.4.3-GCCcore-9.3.0-Python-3.8.2 python-isal/1.1.0-GCCcore-11.3.0
--------------------------------------------- /storage-apps/easybuild/modules/tools ----------------------------------------------
Meson/0.55.1-GCCcore-9.3.0-Python-3.8.2
---------------------------------------------- /storage-apps/easybuild/modules/vis -----------------------------------------------
matplotlib/3.2.1-foss-2020a-Python-3.8.2 wxPython/4.2.0-foss-2021b
Installing your own Python 3.x Environment¶
Creating Python virtual environment stores the files in a current directory, so the first step is to navigate into desired directory:
mkdir ~/virtualenv
cd ~/virtualenv
Let's load version of Python that we wish to clone and will use as basis for our virtual environment (Python/3.10.8-GCCcore-12.2.0 in our case):
module load Python/3.10.8-GCCcore-12.2.0
If you wish to clone either of the default Python2 or Python3, you can skip this step.
Use venv command to create venv_py_310 directory:
python310 -m venv venv_py_310
Virtual environmnent Name
You will probably prefer to use a directory name for your Python environment based on whatever project you are working on rather than venv_py_310. However do not use a name with spaces in it.
Now we need to activate the environment with source command:
source venv_py_310/bin/activate
The shell prompt will now show the name of the Python environment currently active i.e. (venv_py_310). The path to this new Python i.e. ~/venv_py_310/bin will have been added to the start of your existing PATH. It will have also added a “VIRTUAL_ENV” environment variable.
It’s probably a good idea at this stage to make sure that Python’s package manager “pip” is up-to-date:
python -m pip install --upgrade pip
At this point you have working fully customizable Python virtual environment, in which you can install/upgrade/remove any packages you desire. To exit this Python environment use deactivate. This will return you to the default system Python environment.
It can be beneficial to record what packages, and at what versions, are installed in each virtual environment. Running pip freeze command outputs a list of each installed package and its version. This output can then be used to reinstall the packages again into this virtual environment with pip install -r requirements.txt.
source venv_py_310/bin/activate
pip freeze > requirements_venv_py_310.txt
deactivate
If you ever have to reinstall the environemnt simply run:
source venv_py_310_new/bin/activate
pip install -r requirements_venv_py_310.txt
deactivate
Module requirements
If you use one of the Python versions available as modules to create your Python virtual environment you have load that Python version each time you wish to work in your virtual environment.
Using Python 3.x Environment with SLURM scheduler¶
Within a SLURM submission script you will need to activate your Python virtual environment, just the same as you would do at the command line. Add these lines, before the line which runs your python code:
module load Python/3.10.8-GCCcore-12.2.0 # Only if you used Python version from available modules
source $HOME/virtualenv/venv_py_310/bin/activate
and at the end of SLURM submission script add:
deactivate