Singularity¶
Singularity is a container platform. A container is a single file and it is image based. Singularity is an opensource project that was created to run complex applications on HPC clusters. It enables users to have full control of their environment. It allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Singularity on your laptop, and then run it on HPC clusters. Singularity also allows you to leverage the resources of whatever host you are on. This includes HPC interconnects, resource managers, file systems, GPUs and/or accelerators, etc.
A non-privileged user can “swap out” the operating system on the host for one they control. So if the host system is running RHEL6 but your application runs in Ubuntu/RHEL7, you can create an Ubuntu/RHEL7 image, install your applications into that image, copy the image to another host, and run your application on that host in its native Ubuntu/RHEL7 environment.
Singularity containers can be in three different formats:
- read-only squashfs (default) - best for production
- writable ext3 (–writable option)
- writable (ch)root directory (–sandbox option) - best for development
Squashfs and (ch)root directory images can be built from Docker source directly on the cluster, no root privileges are needed. It is strongly recommended to create a native Singularity image to speed up the launch of the container. Singularity is available in modules, add it to your environment using command:
module load singularity
Creating container from scratch¶
To create a container from scratch you need an access to linux operating system running on x86_64 architecture with root privileges and singularity installed. Containers created on other architectures (arm, ppc64 ...) will not work. Start by creating sandbox container from basic ubuntu docker container (or any linux distribution which is available on dockerhub you like) with command as root:
singularity build --sandbox test docker://ubuntu:latest
This will create test folder. To enter the shell of the container use:
singularity shell --writable test
Now your prompt changes to Singularity>. The commands you enter will be executed inside the container. Install your desired software. You can leave the shell using command exit or ctrl+D keyboard shortcut.
For producton use convert the container to readonly squashfs format. This will create one file which is easily transferred to Devana:
singularity build test.sif test/
Use your container on Devana interactively:
singularity shell test.sif
or to run your application inside a container in a script singularity exec test.sif app --arg1 --arg2 file e.g.:
singularity exec test.sif python3 app.py
Creating container from definition file¶
GPU example¶
Tensorflow is commonly used for machine learning projects. The official tensorflow repository on Docker Hub contains NVIDA GPU supporting containers, that will use CUDA for processing.
singularity pull docker://tensorflow/tensorflow:latest-gpu
Commands that run, or otherwise execute containers (shell, exec) can take an --nv option, which will setup the container’s environment to use an NVIDIA GPU and the basic CUDA libraries to run a CUDA enabled application. You can run CUDA application on compute nodes which have GPUs: n[141-148].
OpenMPI example¶
An openmpi example is located in /lustre/home/freeware/EXAMPLE_JOBS/singularity/openmpi/. In this example mpiexec command is executed on singularity container. Inside container is installed the same version of openmpi. Image was built on local linux machine using recipe file openmpi-test.recipe with root privileges:
sudo singularity build openmpi.simg openmpi-test.recipe
During the process of creating the image openmpi with infiniband support is built from source code and simple test is compiled. You can run the example with command sbatch test.sh.
The interactive shell can be invoked by the singularity shell command. This is useful for development purposes. Use the -w | --writable option to make changes inside the container permanent.
A user home directory is mounted inside the container automatically. If you need access to the /scratch local disk storage for your computation, this must be mounted by the -B | --bind option. A complete documentation can be found at the Singularity webpage
A complete documentation can be found at https://sylabs.io/docs/.
Using private docker repository¶
In order to use a private docker repository you have to login first using your credentials using command:
singularity remote login -u username docker://repository_url
You will be prompted for password. This is an equivalent to docker login command. Now you should be able to download containers from the private repository.