Lmod modules¶
Lmod is a Lua based module system that easily handles the $MODULEPATH hierarchical problem. Environment Modules provide a convenient way to dynamically change the user's environment through modulefiles. This includes easily adding or removing directories to the $PATH environment variable.
Detailed (original) documentation on Lmod can be found here.
Useful Commands¶
You can quickly set-up your environment by modules, following commands are useful:
module avail # list all available modules
module load <modulname> # load desired module
module list # list all currently loaded modules
module unload <modulname> # unload desired module
module purge # unload all modules
module use <path_to_modules> # use modules stored in <path_to_modules> folder
module help # display general help
module help <modulname> # display information about module
module keyword <key> # list modules matching the <key> keyword
module spider <key> # similar to keyword command, more effective and precise
Listing Loaded Modules¶
To get an overview of the currently loaded modules, use module list or ml (without specifying extra arguments):
ml
Currently Loaded Modules:
1) EasyBuild/4.7.2
Searching for Available Modules¶
To get an overview of all available modules, you can use module avail or simply ml av:
Availabe modules on Devana
login01:~ $ module avail
R/4.2.3 (D) libvirt/8.10.0 notebook/6.4.10 nvhpc-nompi/23.1 qemu/6.2.0
cuda/12.0.1 maqao/2.17.4 nvhpc-byo-compiler/23.1 nvhpc/23.1 singularity/ce-3.11.0
gcc/12.2 nmap/7.93 nvhpc-hpcx/23.1 openmpi/4.1.4/gcc-12.2 websockify/0.10.0
... ... ...
---------------------------------------- /storage-apps/intel-v.2023/oneapi/modulefiles ----------------------------------------
advisor/latest debugger/2023.0.0 (D) icc32/latest mkl/2023.0.0 (D)
advisor/2023.0.0 (D) dev-utilities/latest icc32/2023.0.0 (D) mkl32/latest
ccl/latest dev-utilities/2021.8.0 (D) init_opencl/latest mkl32/2023.0.0 (D)
ccl/2021.8.0 (D) dnnl-cpu-gomp/latest init_opencl/2023.0.0 (D) mpi/latest
clck/latest dnnl-cpu-gomp/2023.0.0 (D) inspector/latest mpi/2021.8.0 (D)
clck/2021.7.2 (D) dnnl-cpu-iomp/latest inspector/2023.0.0 (D) oclfpga/latest
compiler-rt/latest dnnl-cpu-iomp/2023.0.0 (D) intel_ipp_ia32/latest oclfpga/2023.0.0 (D)
compiler-rt/2023.0.0 (D) dnnl-cpu-tbb/latest intel_ipp_ia32/2021.7.0 (D) tbb/latest
compiler-rt32/latest dnnl-cpu-tbb/2023.0.0 (D) intel_ipp_intel64/latest tbb/2021.8.0 (D)
compiler-rt32/2023.0.0 (D) dnnl/latest intel_ipp_intel64/2021.7.0 (D) tbb32/latest
compiler/latest dnnl/2023.0.0 (D) intel_ippcp_ia32/latest tbb32/2021.8.0 (D)
compiler/2023.0.0 (D) dpct/latest intel_ippcp_ia32/2021.6.3 (D) vpl/latest
compiler32/latest dpct/2023.0.0 (D) intel_ippcp_intel64/latest vpl/2023.0.0 (D)
compiler32/2023.0.0 (D) dpl/latest intel_ippcp_intel64/2021.6.3 (D) vtune/latest
dal/latest dpl/2022.0.0 (D) itac/latest vtune/2023.0.0 (D)
dal/2023.0.0 (D) icc/latest itac/2021.8.0 (D)
debugger/latest icc/2023.0.0 (D) mkl/latest
... ... ...
The string in front of / symbol corresponds to the software name, while the string after / symbol corresponds to software version and/or the compiler toolchain that was used to install the software.
Info
"(D) indicates the default version of the software that will be loaded if no version/toolchain is specified by used in module load command. It is recommended to always use the full module name, as the default values can and will change upon installation of newer versions."
Exploring Modules¶
Let's use different version of Python than the actual old Python2 from system installation.
We can check all available modules by typing module avail and/or module spider. The following reduced list is shown:
Module exploration
login01:~ $ module spider Python
Description:
Python is a programming language that lets you work more quickly and integrate your systems more effectively.
Versions:
Python/2.7.18-GCCcore-10.3.0-bare
Python/2.7.18-GCCcore-11.3.0-bare
Python/3.7.4-GCCcore-8.3.0
Python/3.8.6-GCCcore-10.2.0
Python/3.9.5-GCCcore-10.3.0-bare
Python/3.9.5-GCCcore-10.3.0
Python/3.9.6-GCCcore-11.2.0-bare
Python/3.9.6-GCCcore-11.2.0
Python/3.10.4-GCCcore-11.3.0-bare
Python/3.10.4-GCCcore-11.3.0
Other possible modules matches:
python
We can choose now the desired version of Python, e.g. Python/3.10.4-GCCcore-11.3.0:
module load Python/3.10.4-GCCcore-11.3.0
You can use module commands also in your batch slurm scripts.
Inspecting a Module¶
To see how a module would change the environment, use ml show:
Inspecting module
ml show Python/3.10.4-GCCcore-11.3.0
---------------------------------------------------------------------------------------------------------------------------
/storage-apps/easybuild/modules/lang/Python/3.10.4-GCCcore-11.3.0.lua:
---------------------------------------------------------------------------------------------------------------------------
help([[
Description
===========
Python is a programming language that lets you work more quickly and integrate your systems
more effectively.
More information
================
- Homepage: https://python.org/
Included extensions
===================
alabaster-0.7.12, appdirs-1.4.4, asn1crypto-1.5.1, atomicwrites-1.4.0,
attrs-21.4.0, Babel-2.10.1, backports.entry_points_selectable-1.1.1,
backports.functools_lru_cache-1.6.4, bcrypt-3.2.2, bitstring-3.1.9,
...
webencodings-0.5.1, wheel-0.37.1, xlrd-2.0.1, zipfile36-0.1.3, zipp-3.8.0
]])
whatis("Description: Python is a programming language that lets you work more quickly and integrate your systems
more effectively.")
whatis("Homepage: https://python.org/")
whatis("URL: https://python.org/")
whatis("Extensions: alabaster-0.7.12, appdirs-1.4.4, asn1crypto-1.5.1, atomicwrites-1.4.0, attrs-21.4.0,
Babel-2.10.1, backports.entry_points_selectable-1.1.1, backports.functools_lru_cache-1.6.4, bcrypt-3.2.2,
bitstring-3.1.9, blist-1.3.6, CacheControl-0.12.11, cachy-0.3.0, certifi-2021.10.8, cffi-1.15.0, chardet-4.0.0,
charset-normalizer-2.0.12, cleo-0.8.1, click-8.1.3, clikit-0.6.2, colorama-0.4.4, crashtest-0.3.1,
cryptography-37.0.1, Cython-0.29.28, decorator-5.1.1, distlib-0.3.4, docopt-0.6.2, docutils-0.17.1,
ecdsa-0.17.0, editables-0.3, filelock-3.6.0, flit-3.7.1, flit-core-3.7.1, fsspec-2022.3.0, future-0.18.2,
glob2-0.7, html5lib-1.1, idna-3.3, imagesize-1.3.0, importlib_metadata-4.11.3, importlib_resources-5.7.1,
iniconfig-1.1.1, intervaltree-3.1.0, intreehooks-1.0, ipaddress-1.0.23, jeepney-0.8.0, Jinja2-3.1.2, joblib-1.1.0,
jsonschema-4.4.0, keyring-23.5.0, keyrings.alt-4.1.0, liac-arff-2.5.0, lockfile-0.12.2, MarkupSafe-2.1.1,
mock-4.0.3, more-itertools-8.12.0, msgpack-1.0.3, netaddr-0.8.0, netifaces-0.11.0, packaging-20.9, paramiko-2.10.4,
pastel-0.2.1, pathlib2-2.3.7.post1, pathspec-0.9.0, pbr-5.8.1, pexpect-4.8.0, pip-22.0.4, pkginfo-1.8.2,
platformdirs-2.4.1, pluggy-1.0.0, poetry-1.1.13, poetry-core-1.0.8, psutil-5.9.0, ptyprocess-0.7.0, py-1.11.0,
py_expression_eval-0.3.14, pyasn1-0.4.8, pycparser-2.21, pycrypto-2.6.1, Pygments-2.12.0, pylev-1.4.0,
PyNaCl-1.5.0, pyparsing-3.0.8, pyrsistent-0.18.1, pytest-7.1.2, python-dateutil-2.8.2, pytoml-0.1.21, pytz-2022.1,
regex-2022.4.24, requests-2.27.1, requests-toolbelt-0.9.1, scandir-1.10.0, SecretStorage-3.3.2, semantic_version-2.9.0,
setuptools-62.1.0, setuptools-rust-1.3.0, setuptools_scm-6.4.2, shellingham-1.4.0, simplegeneric-0.8.1,
simplejson-3.17.6, six-1.16.0, snowballstemmer-2.2.0, sortedcontainers-2.4.0, Sphinx-4.5.0, sphinx-bootstrap-theme-0.8.1,
sphinxcontrib-applehelp-1.0.2, sphinxcontrib-devhelp-1.0.2, sphinxcontrib-htmlhelp-2.0.0, sphinxcontrib-jsmath-1.0.1,
sphinxcontrib-qthelp-1.0.3, sphinxcontrib-serializinghtml-1.1.5, sphinxcontrib-websupport-1.2.4, tabulate-0.8.9,
threadpoolctl-3.1.0, toml-0.10.2, tomli-2.0.1, tomli_w-1.0.0, tomlkit-0.10.2, typing_extensions-4.2.0, ujson-5.2.0,
urllib3-1.26.9, virtualenv-20.14.1, wcwidth-0.2.5, webencodings-0.5.1, wheel-0.37.1, xlrd-2.0.1, zipfile36-0.1.3, zipp-3.8.0")
conflict("Python")
load("GCCcore/11.3.0")
...
prepend_path("XDG_DATA_DIRS","/storage-apps/easybuild/software/Python/3.10.4-GCCcore-11.3.0/share")
setenv("EBROOTPYTHON","/storage-apps/easybuild/software/Python/3.10.4-GCCcore-11.3.0")
setenv("EBVERSIONPYTHON","3.10.4")
setenv("EBDEVELPYTHON","/storage-apps/easybuild/software/Python/3.10.4-GCCcore-11.3.0/easybuild/Python-3.10.4-GCCcore-11.3.0-easybuild-devel")
prepend_path("PYTHONPATH","/storage-apps/easybuild/software/Python/3.10.4-GCCcore-11.3.0/easybuild/python")
setenv("EBEXTSLISTPYTHON","wheel-0.37.1,setuptools-62.1.0,pip-22.0.4,blist-1.3.6,pbr-5.8.1,Cython-0.29.28,six-1.16.0,toml-0.10.2,
flit-core-3.7.1,tomli-2.0.1,setuptools_scm-6.4.2,python-dateutil-2.8.2,decorator-5.1.1,liac-arff-2.5.0,pycrypto-2.6.1,ecdsa-0.17.0,
ipaddress-1.0.23,asn1crypto-1.5.1,idna-3.3,pycparser-2.21,cffi-1.15.0,semantic_version-2.9.0,typing_extensions-4.2.0,setuptools-rust-1.3.0,
cryptography-37.0.1,pyasn1-0.4.8,PyNaCl-1.5.0,bcrypt-3.2.2,paramiko-2.10.4,pyparsing-3.0.8,netifaces-0.11.0,netaddr-0.8.0,
mock-4.0.3,pytz-2022.1,bitstring-3.1.9,appdirs-1.4.4,distlib-0.3.4,filelock-3.6.0,zipp-3.8.0,importlib_metadata-4.11.3,
backports.entry_points_selectable-1.1.1,pathspec-0.9.0,pluggy-1.0.0,packaging-20.9,editables-0.3,platformdirs-2.4.1,scandir-1.10.0,
pathlib2-2.3.7.post1,importlib_resources-5.7.1,virtualenv-20.14.1,docopt-0.6.2,joblib-1.1.0,chardet-4.0.0,certifi-2021.10.8,
urllib3-1.26.9,charset-normalizer-2.0.12,requests-2.27.1,xlrd-2.0.1,py_expression_eval-0.3.14,tabulate-0.8.9,ujson-5.2.0,
atomicwrites-1.4.0,py-1.11.0,more-itertools-8.12.0,attrs-21.4.0,backports.functools_lru_cache-1.6.4,wcwidth-0.2.5,iniconfig-1.1.1,
colorama-0.4.4,pytest-7.1.2,MarkupSafe-2.1.1,Jinja2-3.1.2,sphinxcontrib-serializinghtml-1.1.5,sphinxcontrib-websupport-1.2.4,Pygments-2.12.0,
imagesize-1.3.0,docutils-0.17.1,snowballstemmer-2.2.0,alabaster-0.7.12,sphinxcontrib-applehelp-1.0.2,sphinxcontrib-devhelp-1.0.2,
sphinxcontrib-htmlhelp-2.0.0,sphinxcontrib-jsmath-1.0.1,sphinxcontrib-qthelp-1.0.3,Babel-2.10.1,Sphinx-4.5.0,sphinx-bootstrap-theme-0.8.1,
click-8.1.3,psutil-5.9.0,future-0.18.2,sortedcontainers-2.4.0,intervaltree-3.1.0,pytoml-0.1.21,zipfile36-0.1.3,tomli_w-1.0.0,flit-3.7.1,
regex-2022.4.24,intreehooks-1.0,pylev-1.4.0,pastel-0.2.1,crashtest-0.3.1,clikit-0.6.2,jeepney-0.8.0,SecretStorage-3.3.2,keyring-23.5.0,
keyrings.alt-4.1.0,tomlkit-0.10.2,shellingham-1.4.0,requests-toolbelt-0.9.1,pyrsistent-0.18.1,pkginfo-1.8.2,ptyprocess-0.7.0,pexpect-4.8.0,
jsonschema-4.4.0,simplejson-3.17.6,webencodings-0.5.1,html5lib-1.1,cleo-0.8.1,cachy-0.3.0,msgpack-1.0.3,CacheControl-0.12.11,lockfile-0.12.2,
poetry-core-1.0.8,glob2-0.7,poetry-1.1.13,fsspec-2022.3.0,threadpoolctl-3.1.0,simplegeneric-0.8.1")
Loading Modules¶
To load the desired module use module load or ml followed by the name of the module:
Loading Modules
module load Python/3.10.4-GCCcore-11.3.0
module list
Currently Loaded Modules:
1) GCCcore/12.2.0 6) libreadline/8.2-GCCcore-12.2.0 11) libffi/3.4.4-GCCcore-12.2.0
2) zlib/1.2.12-GCCcore-12.2.0 7) Tcl/8.6.12-GCCcore-12.2.0 12) OpenSSL/1.1
3) binutils/2.39-GCCcore-12.2.0 8) SQLite/3.39.4-GCCcore-12.2.0 13) Python/3.10.8-GCCcore-12.2.0
4) bzip2/1.0.8-GCCcore-12.2.0 9) XZ/5.2.7-GCCcore-12.2.0
5) ncurses/6.3-GCCcore-12.2.0 10) GMP/6.2.1-GCCcore-12.2.0
Dependencies
Note that even though we only loaded a single module, the output of module list shows that a whole set of modules was loaded. These are required dependencies for Python/3.10.4-GCCcore-11.3.0.
Conflicting Modules¶
It is important to note that only modules that are compatible with each other can be loaded together. In particular, modules must be installed either with the same toolchain as the modules that are already loaded, or with a compatible (sub)toolchain.
Once you have loaded one or more modules that were installed with the intel/2022b toolchain, all other modules that you load should have been installed with the same toolchain.
Furthermore, only one single version of each software package can be loaded at a given time. For example, once you have the Python/3.10.4-GCCcore-11.3.0 module loaded, you cannot load a different version of Python in the same session/job script, neither directly, nor indirectly as a dependency of another module you want to load.
Unloading Modules¶
To unload any module you can use ms - or ml unload followed by the name of the module:
Unloading Modules
login01:~ $ module load Python/3.10.4-GCCcore-11.3.0
login01:~ $ module list
Currently Loaded Modules:
1) GCCcore/12.2.0 4) bzip2/1.0.8-GCCcore-12.2.0 7) Tcl/8.6.12-GCCcore-12.2.0 10) GMP/6.2.1-GCCcore-12.2.0 13) Python/3.10.8-GCCcore-12.2.0
2) zlib/1.2.12-GCCcore-12.2.0 5) ncurses/6.3-GCCcore-12.2.0 8) SQLite/3.39.4-GCCcore-12.2.0 11) libffi/3.4.4-GCCcore-12.2.0
3) binutils/2.39-GCCcore-12.2.0 6) libreadline/8.2-GCCcore-12.2.0 9) XZ/5.2.7-GCCcore-12.2.0 12) OpenSSL/1.1
login01:~ $ module unload GCCcore/12.2.0
login01:~ $ module list
Currently Loaded Modules:
1) zlib/1.2.12-GCCcore-12.2.0 3) bzip2/1.0.8-GCCcore-12.2.0 5) libreadline/8.2-GCCcore-12.2.0 7) SQLite/3.39.4-GCCcore-12.2.0 9) GMP/6.2.1-GCCcore-12.2.0 11) OpenSSL/1.1
2) binutils/2.39-GCCcore-12.2.0 4) ncurses/6.3-GCCcore-12.2.0 6) Tcl/8.6.12-GCCcore-12.2.0 8) XZ/5.2.7-GCCcore-12.2.0 10) libffi/3.4.4-GCCcore-12.2.0 12) Python/3.10.8-GCCcore-12.2.0
To unload all loaded modules simply issue module purge or ml purge --force.
Saving module selection¶
If you have a set of modules that you load often you can store them in a collection. To store all loaded modules into a ml save <my_favourite_collection>.
If you later with to load these modules simply issue ml restore <my_favourite_collection>.
List of all your collections can be viewed with ml savelist and to inspect the content of collection use ml describe followed by collection name.
To remove a module collection, remove the corresponding entry in $HOME/.lmod.d.