Water and Wastewater Infrastructure Monitoring with SCADA When your business covers hundreds of square miles, the health of your Infrastructure becomes very important.

This is an example of a typical, large water company, serving the water and wastewater needs of over 300,000. On an annual basis, the company manages over 94,000 Acre Feet of water, enough to cover their entire territory under almost 10 inches of water. The system relies on automation spread out over 100s of remote sites, from sewage lift stations to 1000 foot or more deep fresh water wells. This part of the country is far from flat, and the distribution of remote sites requires a vast network of Fiber Optics, wire and radio technology. It is an impressive feat of engineering, and keeping it running efficiently and reliably is the job of automation engineers relying on a collection of controls, Infrastructure components and SCADA technology, all working in harmony day in and day out. When there is a problem, it needs to be diagnosed quickly, to keep the system in balance and to avoid any disruption of service to customers. That is where SNMP comes in. SNMP stands for Simple Network Management Protocol, a standard for monitoring and managing a wide variety of IT (Information Technology) components like Routers, Switches, Wireless Access Points, and Radio Modems. While SNMP is commonly used in the commissioning of an infrastructure, or sporadically as IT Professionals go about their business of supporting their Process Engineer brethren, SNMP is becoming more commonly used in real-time monitoring and management with the automation applications that are responsible for the operation of the core business. For this water company, infrastructure components include BreezeNET® products by Alvarion, Cisco Switches and Routers, and GE MDS Radios and Access Points. These components make up the spider web of connectivity to over 100 remote sites in almost 200 square miles. These products all support SNMP and offer a wealth of information suitable for real-time monitoring. Data within a device is defined by the device vendor in a MIB (Management Information Base) file. Through the Kepware iSNMP (Industrial SNMP) product, water companies can import a device MIB and immediately access the information they need to continuously monitor. In addition to configuration information, devices provide real-time data for network traffic, connection details and diagnostic information. The water company automation system consists of Wonderware's Intouch SCADA solution. Automation engineers have added several HMI Screens (one pictured below) enabling them to monitor their infrastructure both for troubleshooting and historization. SCADA Screen for their Network Overview A variety of data from these devices enables new alarms and decision capabilities. In the screen above, "Green is good". Technical issues are immediately visible by means of a red background. Alarms associated with the iSNMP variables are managed with supplemental InTouch ™ alarms - a major benefit in that additional applications and personnel are reduced, and the alarm management procedures for notification and follow-up wall within standard procedures. Radios and routers are monitored by a standard bandwidth measurement. It is good to see the rate of bytes traveling through I/O ports rather than the percentage of port utilization with respect to capacity of the port. This offers more succinct measurement to be reviewed when planning any network enhancements or corrections. If the bytes transferred ever becomes zero - that is considered an alarm. This calculation is done by doing math on the (iRouterPortThroughtput) value. Doing it this way allows a more understandable way to measure bytes over a selected time period. Usually measured in seconds but calculated every minute or two. The SCADA application has communication fail alarms built in to describe communications to PLCs. However, this does not explain which component in a multi node path communication system requires service. The main screen shows all the network nodes required to be in service to propagate the data through the network. If one of the network nodes downstream of the I/O server becomes non-responsive, it is graphically represented and offers a major clue to how the repairs will be planned. Before this application was created, staff would employ a system of pings, telnet, traceroute, wireshark, Solar Winds and drawings with IP numbers trying to decide which component failed. This process is subtle, tedious and time consuming and requires a good amount of guess work. Monitoring bad communications to PLCs inside Wonderware applications is delivered though a system of running registers and timers. If the running register stops for a parameterized period - say a couple of minutes and alarm is generated. The same concept is applied though the use of the iSystemUptime register of Cisco Routers and the alarm is generated in the Kepware OPC server. This is where the alarm is generated for the Wonderware InTouch ™application to report. This data is historized with the Wonderware InSQL server (Historian) to trend long term comparisons of change over time and to assess when bandwidth will require infrastructure upgrades. Access Point radios can be monitored for both signal strength (RSSI and SNR) and number of connected remotes. Trending signal strength shows the reliability of your connections in varying conditions like weather, tree growth, large trucks parked near antennae or new construction of buildings. The number of connected remotes can display that a site is down, or if there are strange connections or attempts (a possible security concern) or new sites being added without timely notifying support staff. In one instance a trend showing a major bandwidth surge was traced to a development machine with an I/O server with polling parameters set too fast. The monitoring software displayed the surge and technical staff directly monitored the switch to determine the offending IP addressed to solve. The Active Factory Trend tool has worked remarkably well to see the surges as well as declines in network performance. It also serves to date/time stamp network events that can be correlated to physical events to arrive at solutions to problems or at least better describe circumstances that will lead to resolutions of technical issues and problems. There is much more that can be monitored via SNMP. UPS Systems are typically SNMP compliant. Variables such as Voltage, Load, Battery Charge, Battery Life enable you to access the quality of your backup systems, before they are put to the test. Intelligent Power Distribution panels enable SNMP monitoring of voltage, current and load controls on an outlet by outlet basis. These features enable users to perform load management and planning, or reboot remote systems, directly from the automation SCADA system. For Ethernet devices that don't support SNMP, Kepware offers a PING driver, able to connect to a remote device, for example - an IP Based Camera, constantly monitoring its health and making sure it is available when needed. More and more items are Ethernet enabled and tracking their basic health through a Ping, and monitoring response times, is an excellent indication of health and availability. This water company is continually expanding their automation system, including their infrastructure. The addition of infrastructure health monitoring has substantially reduced troubleshooting time and planning their system expansion in a controlled and predictable way. In addition to making their jobs more straight forward, it helps reduce costs, reduce unexpected downtime and enhances their overall system reliability, delivering value to their customers. Return to Kepware Case Studies Page »»