Introduction¶
What IS a software container and what is it good for?¶
A container allows you to stick an application and all of its dependencies into a single package. This makes your application portable, shareable, and reproducible.
Containers foster portability and reproducibility because they package ALL of an applications dependencies.
This means your application won't break when you port it to a new environment. Your app brings its environment with it.
Here are some examples of things you can do with containers:
- Package an analysis pipeline so that it runs on your laptop, in the cloud, and in a HPC environment to produce the same result.
- Publish a paper and include a link to a container with all of the data and software that you used so that others can easily reproduce your results.
- Install and run an application that requires a complicated stack of dependencies.
- Create a pipeline or complex workflow where each individual program is meant to run on a different operating system.
How do containers differ from virtual machines (VMs)?¶
Containers and VMs are both types of virtualization. But it's important to understand the differences between the two and know when to use each.
Virtual Machines install every last bit of an operating system (OS) right down to the core software that allows the OS to control the hardware (called the kernel). This means that VMs:
- Are complete in the sense that you can use a VM to interact with your computer via a different OS.
- Are extremely flexible. For instance you can install a Windows VM on a Mac using software like VirtualBox.
- Are slow and resource hungry. Every time you start a VM it has to bring up an entirely new OS.
Containers share a kernel with the host OS. This means that Containers:
- Are less flexible than VMs. For example, a Linux container must be run on a Linux host OS. (Although you can mix and match distributions.) In practice, containers are only extensively developed on Linux.
- Are much faster and lighter weight than VMs. A container may be just a few MB.
- Start and stop quickly and are suitable for running single apps.
Because of their differences, VMs and containers serve different purposes and should be favored under different circumstances.
- VMs are good for long running interactive sessions where you may want to use several different applications.
- Containers are better suited to running one or two applications, often non-interactively, in their own custom environments.
Docker¶
Docker is currently the most widely used container software. It has several strengths and weaknesses that make it a good choice for some projects but not for others.
philosophy
Docker is built for running multiple containers on a single system and it allows containers to share common software features for efficiency. It also seeks to fully isolate each container from all other containers and from the host system.
Docker assumes that you will be a root user. Or that it will be OK for you to elevate your privileges if you are not a root user.
strengths
- Mature software with a large user community
- Docker Hub
- A place to build and host your containers
- Fully integrated into core Docker
- Over 100,000 pre-built containers
- Provides an ecosystem for container orchestration
- Rich feature set
weaknesses
- Difficult to learn
- Hidden innards
- Complex container model (layers)
- Not architected with security in mind
- Not built for HPC (but good for cloud)
Docker shines for DevOPs teams providing cloud-native micro-services to users.
Singularity¶
Singularity is a container platform. Singularity is an opensource project that was created to run complex applications on HPC clusters. It was developed with security, scientific software, and HPC systems in mind. It enables full control of the environment and allows you to create and run containers that package up pieces of software in a way that is portable and reproducible. You can build a container using Singularity on your laptop, and then run it on HPC clusters. Singularity also allows you to leverage the resources of whatever host you are on. This includes HPC interconnects, resource managers, file systems, GPUs and/or accelerators, etc.
A non-privileged user can “swap out” the operating system on the host for one they control. So if the host system is running RHEL6 but your application runs in Ubuntu/RHEL7, you can create an Ubuntu/RHEL7 image, install your applications into that image, copy the image to another host, and run your application on that host in its native Ubuntu/RHEL7 environment.
Singularity containers can be in three different formats:
- read-only squashfs (default) - best for production
- writable ext3 (–writable option)
- writable (ch)root directory (–sandbox option) - best for development
Squashfs and (ch)root directory images can be built from Docker source directly on the cluster, no root privileges are needed. It is strongly recommended to create a native Singularity image to speed up the launch of the container.
philosophy
Singularity assumes (more or less) that each application will have its own container. It does not seek to fully isolate containers from one another or the host system.
Singularity assumes that you will have a build system where you are the root user, but that you will also have a production system where you may or may not be the root user.
strengths - Easy to learn and use (relatively speaking) - Approved for HPC - Can convert Docker containers to Singularity and run containers directly from Docker Hub - Singularity Container Services! - A place to build and share your containers securely
weaknesses - Younger and less mature than Docker - Smaller user community (as of now) - Under active development (must keep up with new changes)
Singularity shines for scientific software running in an HPC environent.
Downloading and Interacting with Containers¶
You can find pre-built containers in lots of places. Singularity can convert and run containers in many different formats, including those built by Docker.
In this example, we'll be using containers from:
- The Singularity Container Library, developed and maintained by Sylabs
- Docker Hub, developed and maintained by Docker
There are lots of other places to find pre-build containers too. Here are some of the more popular ones:
- Singularity Hub, an early collaboration between Stanford University and the Singularity community
- Quay.io, developed and maintained by Red Hat
- NGC, developed and maintained by NVIDIA
- BioContainers, develped and maintained by the Bioconda group
- Cloud providers like Amazon AWS, Microsoft Azure, and Google cloud also have container registries that can work with Singularity
Downloading containers¶
Singularity is available in modules, add it to your environment using command:
module load singularity
Let's download a container using the pull command.
$ cd ~
$ singularity pull library://godlovedc/funny/lolcow
For now, notice that you have a new file in your current working directory called lolcow_latest.sif
$ ls lolcow_latest.sif
lolcow_latest.sif
This is your container. Or more precisely, it is a Singularity Image Format (SIF) file containing an image of a root level filesystem. This image is mounted to your host filesystem (in a new "mount namespace") and then entered when you run a Singularity command.
Note that you can download the Docker version of this same container from Docker Hub with the following command:
$ singularity pull docker://godlovedc/lolcow
Doing so may produce an error if the container already exists.
Entering containers with shell¶
Now let's enter our new container and look around. We can do so with the shell command.
$ singularity shell lolcow_latest.sif
Depending on the environment of your host system you may see your shell prompt change. Let's look at what OS is running inside the container.
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
No matter what OS is running on your host, your container is running Ubuntu 16.04 (Xenial Xerus)!
Note
In general, the Singularity action commands (like shell, run, and exec) are expected to work with URIs like library:// and docker:// the same as they would work with a local image.
Let's try a few more commands:
Singularity> whoami
trainee01
Singularity> hostname
login01
This is one of the core features of Singularity that makes it so attractive from a security and usability standpoint. The user remains the same inside and outside of the container.
Regardless of whether or not the program cowsay is installed on your host system, you have access to it now because it is installed inside of the container:
Singularity> which cowsay
/usr/games/cowsay
Singularity> cowsay moo
_____
< moo >
-----
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
We'll be getting a lot of mileage out of this silly little program as we explore Linux containers.
This is the command that is executed when the container actually "runs":
Singularity> fortune | cowsay | lolcat
____________________________________
/ A horse! A horse! My kingdom for a \
| horse! |
| |
\ -- Wm. Shakespeare, "Richard III" /
------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
To exit the shell type exit.
Singularity> exit
exit
Executing containerized commands with exec¶
Using the exec command, we can run commands within the container from the host system.
$ singularity exec lolcow_latest.sif cowsay 'How did you get out of the container?'
_______________________________________
< How did you get out of the container? >
---------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
In this example, singularity entered the container, ran the cowsay command with supplied arguments, displayed the standard output on our host system terminal, and then exited.
"Running" a container with (and without) run¶
As mentioned several times you can "run" a container like so:
$ singularity run lolcow_latest.sif
_________________________________________
/ Q: How many Bell Labs Vice Presidents \
| does it take to change a light bulb? A: |
| That's proprietary information. Answer |
| available from AT&T on payment of |
\ license fee (binary only). /
-----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
So what actually happens when you run a container? There is a special file within the container called a runscript that is executed when a container is run. You can see this (and other meta-data about the container) using the inspect command.
$ singularity inspect --runscript lolcow_latest.sif
#!/bin/sh
fortune | cowsay | lolcat
In this case the runscript consists of three simple commands with the output of each command piped to the subsequent command.
Because Singularity containers have pre-defined actions that they must carry out when run, they are actually executable. Note the default permissions when you download or build a container:
$ ls -l lolcow_latest.sif
-rwxr-xr-x 1 trainee01 trainee01 93574075 Feb 28 23:02 lolcow_latest.sif
This allows you to run execute a container like so:
$ ./lolcow_latest.sif
________________________________________
/ It is by the fortune of God that, in \
| this country, we have three benefits: |
| freedom of speech, freedom of thought, |
| and the wisdom never to use either. |
| |
\ -- Mark Twain /
----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Pipes and redirection¶
Singularity does not try to isolate your container completely from the host system. This allows you to do some interesting things. For instance, you can use pipes and redirection to blur the lines between the container and the host system.
$ singularity exec lolcow_latest.sif cowsay moo > cowsaid
$ cat cowsaid
_____
< moo >
-----
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
We created a file called cowsaid in the current working directory with the output of a command that was executed within the container.
We can also pipe things into the container (and that is very tricky).
$ cat cowsaid | singularity exec lolcow_latest.sif cowsay -n
______________________________
/ _____ \
| < moo > |
| ----- |
| \ ^__^ |
| \ (oo)\_______ |
| (__)\ )\/\ |
| ||----w | |
\ || || /
------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
We've created a meta-cow (a cow that talks about cows).
So pipes and redirects work as expected between a container and the host system. If, however, you need to pipe the output of one command in your container to another command in your container, things are slightly more complicated. Pipes and redirects are shell constructs, so if you don't want your host shell to interpret them, you have to hide them from it.
$ singularity exec lolcow_latest.sif sh -c "fortune | cowsay | lolcat"
The above invokes a new shell, but inside the container, and tells it to run the single command line fortune | cowsay | lolcat.
Creating container from scratch¶
To create a container from scratch you need an access to linux operating system running on x86_64 architecture with root privileges and singularity installed. Containers created on other architectures (arm, ppc64 ...) will not work. Start by creating sandbox container from basic ubuntu docker container (or any linux distribution which is available on dockerhub you like) with command as root:
singularity build --sandbox test docker://ubuntu:latest
This will create test folder. To enter the shell of the container use:
singularity shell --writable test
Now your prompt changes to Singularity>. The commands you enter will be executed inside the container. Install your desired software. You can leave the shell using command exit or ctrl+D keyboard shortcut.
For producton use convert the container to readonly squashfs format. This will create one file which is easily transferred to Devana:
singularity build test.sif test/
Use your container on Devana interactively:
singularity shell test.sif
or to run your application inside a container in a script singularity exec test.sif app --arg1 --arg2 file e.g.:
singularity exec test.sif python3 app.py
GPU example¶
Tensorflow is commonly used for machine learning projects. The official tensorflow repository on Docker Hub contains NVIDA GPU supporting containers, that will use CUDA for processing.
singularity pull docker://tensorflow/tensorflow:latest-gpu
Commands that run, or otherwise execute containers (shell, exec) can take an --nv option, which will setup the container’s environment to use an NVIDIA GPU and the basic CUDA libraries to run a CUDA enabled application. You can run CUDA application on compute nodes which have GPUs: n[141-148].
You can also use containers from NVIDIA NGC Catalog, for example version tensorflow 25.02:
singularity pull docker://nvcr.io/nvidia/tensorflow:25.02-tf2-py3
In the following example we will use this image. It also contains Jupyter Notebook, so we will use one of the available tutorials from tensorflow website as the interactive session:
wget https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/keras/classification.ipynb
This jupyter notebook also requires matplotlib. We will add this to the container using the recipe file:
Bootstrap: docker
From: nvcr.io/nvidia/tensorflow:25.02-tf2-py3
%post
python3 -m pip install matplotlib
Save it as tensorflow.recipe.
Warning
You need sudo privileges to build containers from recipe file.
To build the image on your local computer run:
sudo singularity build tensorflow-25.02.sif tensorflow.recipe
Copy the image to the cluster. To run the Jupyter notebook, connect to the Open Ondemand service using your password and 2FA. From the interactive apps select Jupyter Notebook - singularity. In the form specify account you want to use, select gpu partition, 1 node, 16 cores, 1 GPU, 60000 MB of RAM and specify absolute path to the container and click Launch button. This will submit the job to the Slurm scheduler. When the session starts, you will be able to connect to the Jupyter Notebook. In the jupyter notebook session you can then open the classification.ipynb file and run the commands.
OpenMPI example¶
There are two main open-source implementations of MPI at the moment - OpenMPI and MPICH, both of which are supported by SingularityCE.
Note
The MPI in the container must be compatible with the version of MPI available on the host.
The MPI implementation in the container must be carefully configured for optimal use of the hardware if performance is critical.
Here is an example on how to build a container with OpenMPI application from the recipe file on your computer. Your computer should be runnning Linux OS (or use virtual machine) on x86_64 architecture. You can use any OpenMPI version which is installed on Devana. At least the major version must be the same on the cluster and in the container.
Warning
You need sudo privileges to build containers from recipe file.
A simple mpi application source code:
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>
int main (int argc, char **argv) {
int rc;
int size;
int myrank;
rc = MPI_Init (&argc, &argv);
if (rc != MPI_SUCCESS) {
fprintf (stderr, "MPI_Init() failed");
return EXIT_FAILURE;
}
rc = MPI_Comm_size (MPI_COMM_WORLD, &size);
if (rc != MPI_SUCCESS) {
fprintf (stderr, "MPI_Comm_size() failed");
goto exit_with_error;
}
rc = MPI_Comm_rank (MPI_COMM_WORLD, &myrank);
if (rc != MPI_SUCCESS) {
fprintf (stderr, "MPI_Comm_rank() failed");
goto exit_with_error;
}
fprintf (stdout, "Hello, I am rank %d/%d\n", myrank, size);
MPI_Finalize();
return EXIT_SUCCESS;
exit_with_error:
MPI_Finalize();
return EXIT_FAILURE;
}
Bootstrap: docker
From: ubuntu:22.04
%files
mpitest.c /opt
%environment
# Point to OMPI binaries, libraries, man pages
export OMPI_DIR=/opt/ompi
export PATH="$OMPI_DIR/bin:$PATH"
export LD_LIBRARY_PATH="$OMPI_DIR/lib:$LD_LIBRARY_PATH"
export MANPATH="$OMPI_DIR/share/man:$MANPATH"
%post
echo "Installing required packages..."
apt-get update && apt-get install -y wget git bash gcc gfortran g++ make file bzip2 libucx0 libucx-dev
echo "Installing Open MPI"
export OMPI_DIR=/opt/ompi
export OMPI_VERSION=4.1.6
export OMPI_URL="https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-$OMPI_VERSION.tar.bz2"
mkdir -p /tmp/ompi
mkdir -p /opt
# Download
cd /tmp/ompi && wget -O openmpi-$OMPI_VERSION.tar.bz2 $OMPI_URL && tar -xjf openmpi-$OMPI_VERSION.tar.bz2
# Compile and install
cd /tmp/ompi/openmpi-$OMPI_VERSION && ./configure --with-ucx --prefix=$OMPI_DIR && make -j2 install
# Set env variables so we can compile our application
export PATH=$OMPI_DIR/bin:$PATH
export LD_LIBRARY_PATH=$OMPI_DIR/lib:$LD_LIBRARY_PATH
echo "Compiling the MPI application..."
cd /opt && mpicc -o mpitest mpitest.c
To build the container run the following command:
sudo singularity build openmpi.sif openmpi-test.recipe
During the process of creating the container openmpi with infiniband support is built from source code and simple test is compiled. Create a slurm batch script submit.sh:
#!/bin/bash
#SBATCH --partition=short
#SBATCH --job-name=mpitest
#SBATCH --nodes=2
#SBATCH --output=res.txt
#SBATCH --error=err.txt
#SBATCH --ntasks=2
#SBATCH --time=0:05:00
#SBATCH --cpus-per-task=1
#SBATCH --ntasks-per-node=1
module load OpenMPI/4.1.6-GCC-13.2.0
module load singularity
mpirun singularity run openmpi-test.sif /opt/mpitest
sbatch submit.sh
A user home directory is mounted inside the container automatically. If you need access to the /scratch local disk storage for your computation, this must be mounted by the -B | --bind option. A complete documentation can be found at the Singularity webpage
A complete documentation can be found at https://sylabs.io/docs/.
Using private docker repository¶
In order to use a private docker repository you have to login first using your credentials using command:
singularity remote login -u username docker://repository_url
You will be prompted for password. This is an equivalent to docker login command. Now you should be able to download containers from the private repository.