Singularity and Docker | Singularity

The core of a Docker image is basically a compressed set of files, a set of .tar.gz that (if you look in your Docker image folder on your host machine, you will see. We are going to use this local repository for this first set of methods.

The Docker engine communicates with the Docker Hub via the Docker Remote API, and guess what, we can too! The easiest thing to do is create an image, and then pipe a Docker image directly into it from the Docker Registry. This first method does not require having Docker installed on your machine. Lets say that I want to bootstrap tensorflow from Docker. First I should create the tensorflow image:

Note that if you want (much) more detailed output for debugging to the console, you need to enable --verbose mode:

Now I can shell into it, and import tensorflow:

Docker has two commands in the DOCKERFILE that have something to do with execution, CMD and ENTRYPOINT. The differences are subtle, but the best description Ive found is the following:

A CMD is to provide defaults for an executing container.

and

An ENTRYPOINT helps you to configure a container that you can run as an executable.

Given the definition, the ENTRYPOINT is most appropriate for the Singularity %runscript, and so using the default bootstrap (whether from a docker:// endpoint or a Singularity spec file) will set the ENTRYPOINT variable as the runscript. You can change this behavior by specifying IncludeCmd: yes in the Spec file (see below). If you provide any sort of %runscript in your Spec file, this overrides anything provided in Docker. In summary, the order of operations is as follows:

Do a barrel role! Use a spec file! Many times, you want to bootstrap an image, and then either change the %runscript or add additional software or commands in the %post section. To achieve this, you can create a specification file. Currently, these are distributed with the naming format [myfile].def, however (soon) we will use a standard name, Singularity so all specification files can be automatically found. Here is what the spec file would look like for tensorflow:

In the example above, I am overriding any Dockerfile ENTRYPOINT because I have defined a %runscript. If I want the Dockerfile ENTRYPOINT to take preference, I would remove the %runscript section:

Note that the spec file above would be (almost) equivalent to the command:

minus the useless echo at the end. If I want the CMD to take preference, I would add IncludeCmd:

The solutions above would be ideal for saving a custom specification of an image to build at some runtime.

For both import and bootstrap using a build spec file, by default we use the Docker Registry index.docker.io. Singularity first tries the call without a token, and then asks for one with pull permissions if the request is defined. However, it may be the case that you want to provide a custom token for a private registry. You have two options. You can either provide a Username and Password in the build specification file (if stored locally and there is no need to share), or (in the case of doing an import or needing to secure the credentials) you can export these variables to environmental variables. We provide instructions for each of these cases:

You can simply specify your additional authentication parameters in the header with the labels Username and Password:

Again, this can be in addition to specification of a custom registry with the Registry parameter.

You can export your registry, username, and password for Singularity as follows:

If you are having trouble, you can test your token by obtaining it on the command line and putting it into an environmental variable, CREDENTIAL:

This should place the token in the environmental variable TOKEN. To test that your token is valid, you can do the following

The above call should return the tags list as expected.

Finally, we can achieve a shell experience, meaning shelling into Docker image imported into Singularity. We do this by storing the entire image in a temporary location, and then running the same function. You would do something like this:

A common use case is to want to start with a Docker image, possibly add custom commands, and have a Singularity image when you finish. You can read a bit about bootstrapping here to get a sense of adding the custom commands and software. To specify Docker as the build source, you simply need this header:

In the case of omitting the tag (latest) it is assumed that you want the latest image. In the case of omitting the namespace (library) it is assumed that you want the common namespace, library. If you have a reason to use the Docker Engine, we also have a method to do this. The benefit of this method would be that you could use an image built locally (in your local cache) that isnt on Docker Hub.

Here we will access Docker images via the Docker command line tool, meaning using the Docker engine. As is the Docker standard, the image is first looked for in your local cache, and if not found, is pulled from Docker Hub.

We wrapped this entire process into a Docker container itself, which means that you can use a Docker container in a Docker container to export a Docker container into Singularity! Nuts. Full instructions are provided, however here is the gist:

How did this come to be? It so happens that Docker has an export command to pipe this data out, and Singularity has an import command to take them in. Thus, you can do a simple import of a Docker image into a Singularity command by doing:

Where $container_id is the id of a running container obtained with docker ps. However, there are subtle details like the environment and permissions that this method will miss. Its also the case that most Docker images dont run (and stay running) easily unless you do something like:

Early on we created a docker2singularity.sh, a script that you can download and run as follows:

To produce a Singularity image of ubuntu:latest in the present working directory.

Why wont my image bootstrap work? If you cant find an answer on this site, please ping us an issue. If youve found an answer and youd like to see it on the site for others to benefit from, then post to us here.

This entire process will hopefully change in two ways. First, we hope to collapse the image creation and bootstrapping, so you have the option to do them both in one full swing. Second, we hope to eventually figure out some kind of solution to import Docker containers without needing sudo.

Excerpt from:

Singularity and Docker | Singularity