Building Secure Wastewater Management in the Cloud – Automation World

Formally organized in 1834, Waterford Township is located geographically in the center of Oakland County, Michigan, and is home to over 72,000 residents. With 360 miles of water main and 355 miles of sanitary sewer, water management in Waterford is no small task. The Department of Public Works (DPW) operates and maintains 19 production wells, 3 storage tanks, 11 treatment plants, and 63 sewer lift stations.

To run all this, they invested years ago in integrating core applications, including geographic information systems (GIS), asset management systems (AMS), enterprise content management (ECM), and supervisory control and data acquisition (SCADA).

That system delivered a lot of value over the years, but nothing lasts forever.

Time to upgradeIn 2017, Russell Williams, director of public works, and Frank Fisher, engineering superintendent, at Waterford DPW started on a project to upgrade their core SCADA infrastructure. The next year, they attended a conference announcing the release of Opto 22s groov EPIC and were excited by the potential of MQTT Sparkplug to eliminate some long-standing systemic limitations.

With more than 90 controllers on their network, the polling mechanism they used, combined with the limited bandwidth of their radio network, meant that data from each site would update only every 3-5 minutes. Sometimes a lift station would run briefly in between polling cycles, creating gaps in their reporting and inhibiting operators ability to accurately detect issues until alarms eventually made their way through. And for each I/O point they added to the system, this latency only grew worse.

It seemed clear that MQTTs report-by-exception behavior could significantly reduce bandwidth usage and ensure delivery of important system actions.

We have many lift stations that will spend most of their time sitting, Williams explains, [So] why transfer data all the time?

And with no dependence on a central polling program, they saw the possibility to eliminate a systemic bottleneck and potential point of failure.

From proof-of-conceptto productionTo help them execute their vision, Waterford DPW engaged Perceptive Controls, a Michigan-based system integrator specializing in industrial and process control applications for the water/wastewater, food and beverage, and oil and gas industries. But building an MQTT system for the first time came with a learning curve, according to Kevin Finkler, software engineer at Perceptive.

MQTTs publish-subscribe communication model is a definite departure from that of traditional industrial protocols in a few key ways:

After experimenting with a few popular SCADA packages, Perceptive Controls decided on Ignition by Inductive Automation because it offered very tight MQTT integration, including the ability to serve as an MQTT broker itself.

Even though understanding the MQTT communication model took Finkler some work at first, establishing communication was straightforward in the end.

It kind of happens automatically, Finkler says. You basically define a few parameters [in Ignition] to set up the broker. And each of the EPIC devices was pretty simple. You just point it at the broker and it starts sending tags.

I love that both of these sides have embraced MQTT, adds Fisher. It makes the connection seamless.

Previous Configuration: Waterford DPWs legacy infrastructure relied on a network of RTUs and RF transmitters communicating to SCADA workstations in the office.Building defense in depthSeeing an opportunity to leverage cloud computing for greater fault tolerance and scalability, Fisher decided to deploy Ignition directly on Amazon Web Services (AWS), and he and Kevin began building out the mechanisms to secure the new infrastructure.

First, Fisher configured the firewall on AWS to accept traffic only from his groov EPIC controllers and specific Ignition clients in Waterfords and Perceptives offices. Firewalls on the cell modems and EPICs were also configured to accept only trusted IPs.

He then installed a client SSL certificate on each EPIC so that Ignition could authenticate and encrypt the connection, protecting against man-in-the-middle attacks that could expose data or permit unauthorized control.

Every authorized user is required to create strong passwords to access any groov EPIC controller or Ignition client in the system. In addition, every user login is tracked and reported throughout the system as well.

Fisher and Finkler even integrated physical site security into Ignition. Each lift station is secured with an outer door under lock and key, and a physical switch on the door is connected to the local EPIC. Ignition monitors the switch state to detect when someone enters. If a user login is not registered within a specific time with access privileges for that specific room, Ignition then generates a global alarm.

Current Configuration: Waterford DPWs modernized infrastructure publishes data from groov EPIC controllers to a cloud-hosted Ignition SCADA and MQTT broker over a 4G LTE cellular network.Return on investmentAfter completing upgrades on all 63 of its sewage lift stations and six of its clean water sites, the new groov EPIC/Ignition MQTT infrastructure has reduced field updates from multi-minute cycles to sub-second event-driven publications. With that kind of speed, Waterford never misses a system action or alarm notification anymore, and with cell-enabled tablets, operators can stay connected from anywhere through Ignitions mobile-ready HMI client.

Because of MQTTs report-by-exception behavior, in combination with analog I/O deadbanding in each groov EPIC, the new infrastructure has also reduced bandwidth consumption, allowing Waterford to publish even more data than before. They have access to communications and controller diagnosticssuch as update latency, connection time stamps, message size, and firmware versionwhich simply wasnt possible in the old system.

Waterfords cloud-based infrastructure also enables greater flexibility and reliability. If there is ever an issue connecting to the data center in Ohio that hosts the new SCADA server, Fisher can have the entire system up and running in a different data center in 30 minutes. In time, he will likely set up full server redundancy.

In fact, a recent internet outage at the Department of Public Works offices provided an unexpected test of their new system, which kept on working without interruption.

We only lost the old system, says Fisher. Our internal stuff couldnt reach out, of course, but our iPads could connect through Verizon... and I was able to get back in touch. In a situation like this, the old system couldnt send out alarms because it depended on a local connection. The new system didnt even notice or care because its not running anything local.

More to comeWith huge increases in bandwidth, the low administrative overhead of MQTT Sparkplug, and EPICs providing spare data processing at the edge, Waterford can continue expanding its system for a very long time. Each new device or application they add only needs a connection to the MQTT broker to produce or consume data for/from the whole system.

We are still trying to figure out what else we can do with this, says Fisher. We have a lot of other instrumentation that we want to be able to pull data from out in the field that wasnt really feasible before not just at our lift stations and our treatment plants but throughout the organization. Where can we use [MQTT] with flowmeters? Where can we use it throughout all of our assets to give us a better overview? Were just beginning that journey.

For more information, visit http://www.perceptivecontrols.com or contact Frank Fisher at ffisher@waterfordmi.gov.

See the rest here:

Building Secure Wastewater Management in the Cloud - Automation World