Singularity Tutorial – SDSC

Background

What is Singularity?*

"Singularity enables users to have full control of their environment. Singularity containers can be used to package entire scientific workflows, software and libraries, and even data. This means that you dont have to ask your cluster admin to install anything for you - you can put it in a Singularity container and run."

There are numerous good tutorials on how to install and run Singularity on Linux, OS X, or Windows so we won't go into much detail on that process here. In this tutorial you will learn how to run Singularity on Comet. First we will review how to access a compute node on Comet and provide a simple example to help get you started. There are numerous tutorial on how to get started with Singularity, but there are some details specific to running Singularity on Comet which are not covered in those tutorials. This tutorial assumes you already have an account on Comet. You will also need access to a basic set of example files to get started. SDSC hosts a Github repository containing a Hello world! example which you may clone with the following command:

> git clone https://github.com/hpcdevops/singularity-hello-world.git

Below is a typical list of commands you would need to issue in order to implement a functional Python installation for scientific research:

COMMAND=apt-get -y install libx11-dev COMMAND=apt-get install build-essential python-libdev COMMAND=apt-get install build-essential openmpi-dev COMMAND=apt-get install cmakeCOMMAND=apt-get install g++ COMMAND=apt-get install git-lfs COMMAND=apt-get install libXss.so.1COMMAND=apt-get install libgdal1-dev libproj-dev COMMAND=apt-get install libjsoncpp-dev libjsoncpp0 COMMAND=apt-get install libmpich-dev --userCOMMAND=apt-get install libpthread-stubs0 libpthread-stubs0-dev libx11-dev libx11-d COMMAND=apt-get install libudev0:i386COMMAND=apt-get install numpy COMMAND=apt-get install python-matplotlib COMMAND=apt-get install python3

Singularity allows you to avoid this time-consuming series of steps by packaging these commands in a re-usable and editable script, allowing you to quickly, easily, and repeatedly implement a custom container designed specifically for your analytical needs.

The diagram below compares a VM vs. Docker vs. Singularity.

Next, let's get some hands-on experience with Singularity. The following tutorial includes links to asciinema video tutorials created by SDSC HPC Systems Manager, Trevor Cooper (Thanks, Trevor!) which allow you to see the console interactivity and output in detail. Look for the video icon like the one shown to the right corresponding to the task you are currently working on.

First we download and upack the source using the following commands (assuming your user name is 'test_user' and you are working on your local computer with super user privileges):

[test_user@localhost ~]$ wget https://github.com/singularityware/singularity/releases/download/2.5.1/singularity-2.5.1.tar.gz tar -zxf singularity-2.5.1.tar.gz

If the file is successfully extracted, you should be able to view the results:

Next we configure and build the package. To configure, enter the following command (we'll leave out the command prompts):

To build, issue the following command:

This may take several seconds depending on your computer.

To complete the installation enter:

You should be prompted to enter your admin password.

Once the installation is completed, you can check to see if it succeeded in a few different ways:

You can also run a selftest with the following command:

The output should look something like:

The process of building a Singularity container consists of a few distinct steps as follows.

We will go through each of these steps in detail.

We recommend building containers using the same version of Singularity, 2.5.1, as exists on Comet. This is a 2 step process.

Step 1: run the script below to remove your existing Singularity:

#!/bin/bash## A cleanup script to remove Singularity

sudo rm -rf /usr/local/libexec/singularitysudo rm -rf /usr/local/etc/singularitysudo rm -rf /usr/local/include/singularitysudo rm -rf /usr/local/lib/singularitysudo rm -rf /usr/local/var/lib/singularity/sudo rm /usr/local/bin/singularitysudo rm /usr/local/bin/run-singularitysudo rm /usr/local/etc/bash_completion.d/singularitysudo rm /usr/local/man/man1/singularity.1

Step 2: run the following script to install Singularity 2.5.1:

#!/bin/bash## A build script for Singularity (http://singularity.lbl.gov/)

declare -r SINGULARITY_NAME='singularity'declare -r SINGULARITY_VERSION='2.5.1'declare -r SINGULARITY_PREFIX='/usr/local'declare -r SINGULARITY_CONFIG_DIR='/etc'

sudo apt updatesudo apt install python dh-autoreconf build-essential debootstrap

cd ../tar -xzvf "${PWD}/tarballs/${SINGULARITY_NAME}-${SINGULARITY_VERSION}.tar.gz"cd "${SINGULARITY_NAME}-${SINGULARITY_VERSION}"./configure --prefix="${SINGULARITY_PREFIX}" --sysconfdir="${SINGULARITY_CONFIG_DIR}"makesudo make install

To create an empty Singularity container, you simply issue the following command:

This will create a CentOS 7 container with a default size of ~805 Mb. Depending on what additional configurations you plan to make to the container, this size may or may not be big enough. To specify a particular size, such as ~4 Gb, include the -s parameter, as shown in the following command:

To view the resulting image in a directory listing, enter the following:

Next, we will import a Docker image into our empty Singularity container:

Once the container actually contains a CentOS 7 installation, you can shell into it with the following:

Once you enter the container you should see a different command prompt. At this new prompt, try typing:

Your user id should be identical to your user id outside the container. However, the operating system will probably be different. Try issuing the following command from inside the container to see what the OS version is:

Next, lets trying writing into the container (as root):

You should be prompted for your password, and then you should see something like the following:

Next, lets create a script within the container so we can use it to test the ability of the container to execute shell scripts:

The above command assumes you know the vi editor. Enter the following text into the script, save it, and quit the vi editor:

You may need to change the permissions on the script so it can be executable:

Try running the script manually:

The output should be:

Bootstrapping a Singularity container allows you to use what is called a definitions file so you can reproduce the resulting container configurations on demand.

Lets say you want to create a container with Ubuntu, but you may want to create variations on the configurations without having to repeat a long list of commands manually. First, we need our definitions file. Below is the contents of a definitions file which should suffice for our purposes.

%runscriptexec echo The runscript is the containers default runtime command!

%files/home/testuser/ubuntu.def /data/ubuntu.def

%labelsAUTHOR testuser@sdsc.edu

%postapt-get update && apt-get -y install python3 git wgetmkdir /dataecho The post section is where you can install and configure your container.

To bootstrap your container, first we need to create an empty container.

Now, we simply need to issue the following command to configure our container with Ubuntu:

This may take a while to complete. In principle, you can accomplish the same result by manually issuing each of the commands contained in the script file, but why do that when you can use bootstrapping to save time and avoid errors.

If all goes according to plan, you should then be able to shell into your new Ubuntu container.

Of course, the purpose of this tutorial is to enable you to use the San Diego Supercomputer Centers Comet supercomputer to run your jobs. This assumes you have an account on Comet already. If you do not have an account on Comet and you feel you can justify the need for such an account (i.e. your research is limited by the limited compute power you have in your government-funded research lab), you can request a Startup Allocation through the XSEDE User Portal:

https://portal.xsede.org/allocations-overview#types-trial

You may create a free account on the XUP if you do not already have one and then proceed to submit an allocation request at the above link.

[NOTE: SDSC provides a Comet User Guide ( http://www.sdsc.edu/support/user_guides/comet.html ) to help get you started with Comet. Learn more about The San Diego Supercomputer Center at http://www.sdsc.edu .]

This tutorial walks you through the following four steps towards running your first Singularity container on Comet:

Once you have created your container on your local system, you will need to transfer it to Comet. There are multiple ways to do this and it can take a varying amount of time depending on its size and your network connection speeds.

To do this, we will use scp (secure copy). If you have a Globus account and your containers are more than 4 Gb you will probably want to use that file transfer method instead of scp.

Browse to the directory containing the container. Copy the container to your scratch directory on Comet. By issuing the following command:

The container is ~805 Mb so it should not take too long, hopefully.

Once the file is transferred, login to Comet (assuming your Comet user is named 'test_user'):

Navigate to your scratch directory on Comet, which should be something like:

Next, you should submit a request for an interactive session on one of Comets compute, debug, or shared nodes.

Once your request is approved your command prompt should reflect the new node id.

Before you can run your container you will need to load the Singularity module (if you are unfamiliar with modules on Comet, you may want to review the Comet User Guide). The command to load Singularity on Comet is:

You may issue the above command from any directory on Comet. Recall that we added a hello_world.sh script to our centos7.img container. Lets try executing that script with the following command:

If all goes well, you should see Hello, World! in the console output. You might also see some warnings pertaining to non-existent bind points. You can resolve this by adding some additional lines to your definitions file before you build your container. We did not do that for this tutorial, but you would use a command like the following in your definitions file:

You will find additional examples located in the following locations on Comet:

and

It is best to avoid working on Comets login nodes since they can become a performance bottleneck not only for you but for all other users. You should rather allocate resources specific for computationally-intensive jobs. To allocate a compute node for your user on Comet, issue the following command:

This allocation requests a single node (-N 1) for a total time of 10 minutes (-t 00:10:00). Once your request has been approved, your computer node name should be displayed, e.g. comet-17-12.

Now you may login to this node:

Notice that the command prompt has now changed to reflect the fact that you are on a compute node and not a login node.

Next, load the Singularity module, shell into the container, and execute the hello_world.sh script:

If all goes well, you should see Hello, World! in the console output.

Of course, most users simply want to submit their jobs to the Comet queue and let it run to completion and go on to other things while waiting. Slurm is the job manager for Comet.

Below is a job script (which we will name singularity_mvapich2_hellow.run) which will submit your Singularity container to the Comet queue and run a program, hellow.c (written in C using MPI and provided as part of the examples with the mvapich2 default installation).

#!/bin/bash #SBATCH --job-name="singularity_mvapich2_hellow" #SBATCH --output="singularity_mvapich2_hellow.%j.out" #SBATCH --error="singularity_mvapich2_hellow.%j.err" #SBATCH --nodes=2 #SBATCH --ntasks-per-node=24 #SBATCH --time=00:10:00 #SBATCH --export=all module load mvapich2_ib singularity

CONTAINER=/oasis/scratch/comet/$USER/temp_project/singularity/centos7-mvapich2.img

mpirun singularity exec ${CONTAINER} /usr/bin/hellow

The above script requests 2 nodes and 24 tasks per node with a wall time of 10 minutes. Notice that two modules are loaded (see the line beginning with module), one for Singularity and one for MPI. An environment variable CONTAINER is also defined to make it a little easier to manage long reusable text strings such as file paths.

You may need to add a line specifying with allocation to be used for this job. When you are ready to submit the job to the Comet queue, issue the following command:

To view the status of your job in the Comet queue, issue the following:

When the job is complete, view the output which should be written to the output file singularity_mvapich2_hellow.%j.out where %j is the job ID (lets say the job ID is 1000001):

See the original post:

Singularity Tutorial - SDSC