The Evolution and Maturation of HPC in the Enterprise – CIO

By Adnan Khaleel

The convergence of high performance computing (HPC) and big data has been under way for years. As I noted in an earlier blog, HPC and big data grew up in different worlds and are now coming togetherdue to necessity. People using HPC applications often work with big data, and people working with big data often need the processing power of HPC systems. This convergence is giving rise to the era of high performance data analytics (HPDA) in the enterprise.

Lets take a step back. For enterprises, data is coming at much faster rates than anyone had expected. Whether its from the Internet of Things, webpages, commercial transactions or other sources, the amount of data pouring into enterprise data centers exceeds current storage capacity. This flood of data creates a new class of data consolidation, data handling and data management challenges. Organizations cant just let the data pile up. They now need to make deliberate decisions about what data to store, what data to analyze and what data to discard.

Above all, enterprises need to find ways to turn the flood of data into meaningful insights. This process increasingly requires HPC capabilities that make applications run as fast as possible. In many cases, enterprises need to generate insights in real timewhether they need to optimize the performance of remote equipment, respond faster to a customers needs or put the brakes on a potentially fraudulent transaction.

Lets take the example of the many enterprises that are getting hit with an ever-growing wave of data from our world of connected devices, the Internet of Things (IoT). To capitalize on this data, whether in real time or over a period of time, enterprises need to apply sophisticated machine learning and deep learning techniques, and these techniques require HPC systems paired with big data platforms and data analytics tools.

With HPDA, enterprises use HPC technologies to analyze big data for rapid insights, real-time results and predictive analytics. One study found that 67 percent of HPC users are already doing HPDA, in addition to or instead of traditional HPC.[1]

While HPDA is needed in traditional research-driven applications of HPC, it is becoming a must-have in enterprise environments. Depending on the industry, an enterprise might need to leverage data-centric HPC platforms for more traditional HPC applications like genomics, financial modeling and signal processing, as well as new and emerging HPDA applications like personalized medicine, fraud detection and machine learning.

The rise of new tools and technologies

For organizations that need HPDA, there is good news on the technology front: The tools and technologies for merging HPC with data analytics are maturing rapidly. Better still, HPC and big data platforms are converging in a manner that reduces the need to move data back and forth between HPC and storage environments. This convergence helps organizations avoid a great deal of overhead and latency that comes with disparate systems.

Today, organizations can choose from a rapidly growing range of tools and technologies like streaming analytics, graph analytics, and exploratory data analysis in HPC environments. Lets take a brief look at these tools.

HPDA in action: case studies

Lets consider a couple of real-life examples of HPDA in action. These examples show how companies are capitalizing on the convergence of technologies for HPC and big data.

To help fight cancer and other diseases, TGen needed extremely scalable, reliable and available HPC nodes to develop personalized medical treatments. To meet this need, TGen optimized its infrastructure, scaling its existing Dell EMC HPC cluster with Dell EMC PowerEdge blades. The system incorporates powerful big data and analytics tools, leveraging a Dell EMC Hadoop platform and Statistica software. The increased performance helps TGen accelerate results, enabling researchers to expand treatments to a larger number of patients. Watch the video.

Another Dell EMC customer, Sensus, needed to increase its data set size to be able to more easily visualize meter sensor performance problems. To meet this need, the company implemented a data cluster and a data lakebased on a Hadoop platform and technologies from Dell EMC and Intelthat consolidates manufacturing, testing and other data streams. With this consolidated platform, Sensus can quickly analyze data from 17 million gas, electric and water meter sensors, and proactively identify device problems, helping to predict and prevent future device failures. Read the case study.

Enabling proactive maintenance with HPDA

On the IoT front, HPDA technologies are enabling predictive maintenance of assets to help prevent equipment failures, extend machine life and help organizations gain a better return on their assets. These technologies go beyond condition monitoring to enable condition understanding. On its own, condition monitoring provides time to act, but when data is dynamically provided to a device-specific predictive model you can achieve condition understanding. That means your users will have time to act on maintenance events and have a clear understanding of the actions they need to take.

For organizations new to IoT, the challenges are numerous, spanning both hardware and software. For example, they need to:

And this is where expertise comes in extremely handy. With that thought in mind, Dell EMC has joined forces with Software AG and Kepware to produce an end-to-end solution for proactive maintenance. It offers the complete hardware-software stack that easily allows for the management of IoT sensors, the data produced, and the analysis of that data in real timeultimately easing the deployment of a comprehensive IoT based solution for infrastructure maintenance.

Thats just one of countless advances made possible by the rise of technologies and solutions for high performance data analytics. For a look at more of these technologies and solutions, visit Dell.com/HPC.

Adnan Khaleel is a Global Sales Strategist for Dell EMC.

[1] HPCwire, IDC 2015. The Changing Face of HPC.

Originally posted here:

The Evolution and Maturation of HPC in the Enterprise - CIO