Greg Kurtzer invented software called Singularity to enable the use of containers in high-performance computing (Credit: Marillyn Chung/Berkeley Lab)

Scientists used to come to Gregory Kurtzer of Lawrence Berkeley National Laboratorys (Berkeley Labs) IT department a lot, asking for a better way to use software containers in a high-performance computing (HPC) environment. After a while he got tired of saying, Sorry, not possible. So he invented a solution and named it Singularity.

Within a few months of its release last year, Singularity took off. Computing-heavy scientific institutions worldwidefrom Stanford University to the Massachusetts Institute of Technology to various sites on the European Grid e-Infrastructureflocked to the software. Singularity was also recently recognized by HPCwire editors as one of five new technologies to watch.

Singularity has been making huge strides in the computing community, Kurtzer said, with some surprise, adding that Open Science Grid, a consortium that provides distributed computing resources for scientific research, has served over 20 million containers with Singularity.

Its now on its seventh release (version 2.2.1) and has caught on so quickly that Kurtzer has launched SingularityWare LLC to further develop and support the open-source software. The company is being funded by RStor Inc., a startup based in Saratoga, California. Kurtzer, the long-time technical lead and architect for the HPC Services group at Berkeley Lab with a joint appointment at UC Berkeley, will shift to an advisory role at the Lab in order to focus on Singularity.

Berkeley Lab makes some Lab-developed software available at no cost to maximize its impact and to participate in the open-source software community, said ElsieQuaite-Randall, Berkeley Labs Chief Technology Transfer Officer. Singularity fosters innovation as open-source software, and now SingularityWare LLClike other Berkeley Lab startupswill set out to expand the reach and adoption of an important technology.

A typical case where users might need Singularity is if they want to run an application such as Googles TensorFlow. They may need a very specific version of Tensor Flow installed, Kurtzer said. They can create a container to do that in about five minutes. Then they can take that container, bring it to our environment and run it, even if we dont have that version of Tensor Flow installed.

Software containers make it possible to take your entire computing environment, including your files and all the applications you want to run, and encapsulate it so it can be easily replicated on another machine without worrying whether the new machine has a compatible operating system, libraries, applications, and so forth.

Containers share some of the use cases of virtual machines but without the code redundancy and performance hit associated with virtualization, Kurtzer said. Singularity containers allow a user to encapsulate an entire OS (operating system) environment and use it on a shared HPC system like any other program, without an admin doing anything.

Another example where Singularity would be useful would be allowing other scientists to reproduce experiments. Say you just published an article. Wouldnt it be nice to have a location you can cite where someone can download the Singularity container and replicate all the experiments? Kurtzer asked. Someone can enter the container, and now theyre sitting in the exact same environment as you were.

Containerization was developed for enterprise environments, where it has become very popular, especially with the rise of Dockers container technology. Dockers container solution is for the enterprise. But the scientific use case is quite different, Kurtzer said. Our goal isnt to run as many containers as we can on a single host, with each having the illusion of sole occupancy and isolation, but to run maybe one, and enable it to utilize all the resources on that host. Its kind of the opposite of isolation!

So Kurtzer started working on his own solution, and four months later, the first version was released last spring. When I started working on it, I asked, what do scientists really need from containers? They need reproducibility, mobility, and also freedomthe ability to install their own applications and run in their own environment, and store it just like any other data file, Kurtzer said. Thats what Singularity solves for scientific computing.

Kurtzer chose the name Singularity for its meaning in astronomy. As I understand it, its the culmination of a whole bunch of matter in the universe forming a single infinitely dense point, he said. Thats what I was thinking when I was creating Singularitytaking everything necessary to create a reproducible scientific environment and putting it in one file.

Singularity also enables users to run legacy workflows easily. Kurtzer cites one example of how his group saved an 18-year-old workflow from failing hardware and was able to convert it to a Singularity container that is still being used in production today.

Kurtzer believes Singularity will benefit scientists who may not even know they need it. Were trying to reach out to more scientists and engage with additional groups, especially those who are not traditional HPC users, also known as the computational long tail of science, Kurtzer said. We have a lot of users that are running computationally intensive jobs on their laptops and workstations and not making use of the dedicated computational cycles that are designed specifically for computing and available to them. With Singularity we can easily make these large computing resources tangible.

# # #

Lawrence Berkeley National Laboratory addresses the worlds most urgent scientific challenges by advancing sustainable energy, protecting human health, creating new materials, and revealing the origin and fate of the universe. Founded in 1931, Berkeley Labs scientific expertise has been recognized with 13 Nobel Prizes. The University of California manages Berkeley Lab for the U.S. Department of Energys Office of Science. For more, visit http://www.lbl.gov.

DOEs Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.

More here:

Berkeley Lab's Open-Source Spinoff Serves Science | Berkeley Lab - Lawrence Berkeley National Laboratory