The Software Behind IBM's Watson Will Be Probing The Mysteries Of Physics

Scientists at the PETRA III particle accelerator. (Credit: IBM/DESY)

Last week, the Deutsches Elektronen-Synchrotron one of the largest research centers in Germany announced that it has entered into a collaboration with IBM to enhance its computers abilities to crunch the massive amounts of data being generated every day by its particle accelerators.

DESY has a variety of instruments being used every day by teams of scientists all over the world. These researchers are exploring subatomic physics, astrophysics, chemistry, biology and a host of other different types of sciences. The amount of data that the research institute generates is staggering. To give you an example, just one instrument currently being built, the X-ray laser European XFEL, will be producing about 100 Petabytes of data per year when its complete. And thats just one instrument.

To handle the data volume, IBM IBM will be supplying DESY with its Elastic Storage system. This is the hardware behind Watson and IBMs major supercomputing systems that IBM began releasing as a general product in May of this year. When the system is installed at DESY, it will be able to handle 20GB per second of data and be quickly accessible to who needs it.

This system makes data available at any time around the world, Dr. Volker Glzow, head of DESY IT told me. Its basically at your fingertips.

The Elastic Storage system, which is part of what Watson used to plow through hundreds of millions of Wikipedia articles to prep for Jeopardy, is a hardware-agnostic software layer that uses proprietary algorithms to to scale out and process data along thousands of nodes. But its speed comes from a unique ability it doesnt need multiple copies of data to work.

Most storage systems require multiple copies of data to ensure accessibility, Jamie Thomas, IBMs General Manager Storage and Software Defined Systems told me. Elastic Storage includes a unique invention that allows it to recreate data should something happen to the single copy.

This is appealing to a lot of scientific institutions, Thomas told me, because until recently most storage solutions like this were built in house. But with so much data being produced by modern instruments, labs are overwhelmed.

Its also a benefit, Glzow pointed out to me, because it allows teams to do work at DESY from anywhere in the world. They can submit a proposal, have the experiment performed, then have the data delivered in a manner thats quick and easy to access.

The system is currently scheduled to go live in March of 2015 after some upgrades to a particle accelerator are completed.