The Lower Colorado River Authority (LCRA), headquartered in Austin, plays a variety of important roles in Central Texas: delivering electricity, managing the water supply and environment of the lower Colorado River basin, planning and coordination of water and wastewater needs, providing public recreation areas, and supporting community and economic development in 58 counties.
LCRA provides low-cost electricity to more than 40 retail utilities, including cities and electric cooperatives and operates more than 3,300 miles of transmission lines statewide. It operates six dams on the Colorado River, regulates water discharges to manage floods, and releases water for sale to municipal, agricultural and industrial users. Protecting water quality in the lakes and river is also a vital part of LCRA's mission.
The Challenge
Supporting the mission of LCRA to the communities it serves, is the organization's main Data Center. While there are several SCADA applications running onsite, one of the main systems that serve at the heart of LCRA is the Facilities Monitoring System (FMS) in Austin. The FMS began as a simple environmental monitoring system but it just keeps growing and expanding, according to Mike Lieblich, SCADA Systems Integrator for LCRA Corporate Services. The FMS is responsible for monitoring the main General Office Complex Data Center in Austin as well as three other smaller centers across town.
"Our challenge was to build a solution that would allow us to gain maximum visibility into our Data Center sites to manage the environmental issues associated with the centers — temperature, humidity, air flow — which were previously being managed manually. When we had a temperature issue, we didn't always know right away and actually had to send a person in to remedy the problem."
As more and more demands were put on the Data Center, such as the addition of servers, and control room air conditioners (CRAC's), LCRA recognized they needed a technologically advanced, robust system that could provide visibility to help efficiently manage the Data Center environments.
Finding the Solution
Although there are many software programs that gather and display SNMP data, LCRA needed a solution to bring in not only SNMP data but also Modbus TCP and RTU data in addition to PLC data into a single application. "The implementation team was aware of the advantages that iSNMP could bring to us but to our knowledge no one at our company had actually worked with this kind of solution before," Mike said.
"No one was really sure what we wanted or needed," Mike noted. "It was all new territory." Since LCRA was using Wonderware InTouch throughout the company they decided to look for a Wonderware solution that would integrate easily into their existing environment. "We found that Kepware could provide the iSNMP technology and that it would give us the valuable insight we needed into the performance of our control system network. We also learned that Kepware's iSNMP driver integrated seamlessly with Wonderware's InTouch product which LCRA has used for many years."
Once the facilities management team quantified the benefits that they felt could bring the needed cost-savings and efficiencies, the plan was rolled out and continues to be upgraded as the Data Centers grow. "Since the time the iSNMP solution was implemented, we have at least tripled the size of the InTouch SCADA application and don't see any end in sight," Mike added.
Putting the Pieces Together to Boost the Bottom Line
The FMS application is currently monitoring environmental data, temperatures, humidity, fluid detection, rack power consumption in the Data Centers to LCRA's generators, PDU's, UPS's and Control Room Air Conditioning (CRAC) units. Cameras are placed throughout the data center and are accessed through the FMS SCADA. The FMS utilizes the Wonderware Historian to "historize" all data points that are stored in a SQL database. This data is available to the users through the Active Factory client interface. Active Factory gives the user the ability explore data graphically, trend data points, use the process analysis feature and create ad hoc reports as well as scheduled reports. The FMS utilizes the Wonderware alarm system to actively monitor all critical devices. "An additional program called SCADAlarm is used to alert the users via a text message or email upon activation of any programmed alarm and when the alarm is cleared," Mike explained.
The Data Center contains many rows of server racks. Mike has been told that LCRA, utilizing Kepware's iSNMP driver, is one of a very few organizations that have actually pursued monitoring of power from individual power strips on all the racks. LCRA's System Integration team monitors information on several servers and is able to look at different data centers locations across Austin. The focus is environmental conditions and power. "This is the most interesting application of the Kepware iSNMP. We receive a large amount of SNMP data - we know what power loads are - status of all monitored equipment and devices - even how much UPS battery time we have left during an outage. We have a one line screen programmed into the FMS SCADA showing power flow from the incoming utility into the data center. This one line shows the Automatic Transfer Switch positions and where the power is coming from whether it is from the generator or utility. This is critical information needed during a power outage," Mike added.
LCRA also added RFID monitoring capabilities to the FMS and according to Mike, "this technology at all our sites is going to expand quite a bit." Currently there is an RFID temperature and humidity sensor in almost every other rack in the Data Centers. The sensors are active RFID tags that beacon temperature or humidity data every 10 seconds to a wireless "reader." That reader communicates to the RFCode Sensor Manager program storing the data in Modbus registers where the SCADA polls the data via Modbus TCP communication. RFCode located in Austin just released the Modbus version of their Sensor Manager software. The System Integration team has been beta testing this for over a month and it has worked without a hitch and should be put into production in mid-September. There are also plans to implement RFID dry contact and fluid detection sensors in the future.
In addition to RFID and Kepware's iSNMP, which has worked "perfectly", Mike says, the FMS also utilizes Modbus RTU communication to bring data into the Wonderware InTouch application. Modbus TCP however allows FMS to connect the field devices directly to the network via wireless or Cat 5 cable.
"This year we will be installing revenue grade KW meters to monitor power for LCRA's Energy Star program," Mike noted. "The KW meters will communicate over the wireless network utilizing Modbus TCP. We anticipate monitoring close to 40 sites."
"This is the FMS — many people can view this application via a terminal service connection. It allows many users to connect to the FMS terminal server where they run a separate instance of the FMS SCADA application on the server to view the system data. So it's pretty quick. It allows a lot of people to look at the real time data through a connection over the network or offsite through a VPN," Mike added.
The System Integration team credits Kepware's iSNMP, with being a critical component of the system solution. "We wouldn't be able to access a lot of data if we weren't using Kepware's iSNMP driver. Right now we collect environmental data two different ways - with RFID sensors and Kepware. Kepware's iSNMP is extremely valuable to the IT department for monitoring environmental conditions, servers and equipment. We couldn't see most of that data without iSNMP since many of the devices only provide SNMP data. Every server rack has an APC power strip and we monitor the power on most of the racks via iSNMP," Mike explained. "With this amount of power and temperature data we are able to monitor the power load on each rack and adjust room temperature set points to a higher level. "
"Configuring the iSNMP driver is an easy process and once enabled, it works flawlessly … set it and forget it," he added.
With Kepware's iSNMP, the FMS has grown to monitor well over 100 server racks not to mention the CRAC units, and UPS's. "We also utilize the iSNMP ping driver to monitor communication to all the SNMP enabled devices and equipment," Mike said.
Mike also claims the use of iSNMP as helping create a collaborative environment with IT. "It's not very common to have SCADA groups interfacing successfully with the IT department. We have a good relationship with IT for many reasons but it's even better now that we can deliver needed SNMP information via this program. I've never worked as closely with IT before and I've been doing this a long time. One of the great things about this FMS system is that we can bring in IT specific information via SNMP and industrial information- say from a generator, power distribution system or a PLC via other protocols - you can bring that all into a single application and it gives great visibility to anyone that needs access to this data."
"Using Wonderware with iSNMP, RFID and Modbus protocols, we got the granular visibility to effectively monitor all the data points," Mike explained.. "It's allowing us to monitor data center temperatures where we can now more efficiently direct air flow and raise temperature set points according to ASHRE guidelines (American Society of Heating, Refrigerating and Air-Conditioning Engineers). That saves us money right there." Mike also attributes the system and the visibility it gives to the SYSTEM INTEGRATION team and IT teams, for eliminating the need for any professional consulting services. "We don't need a consultant to tell us how to manage temperature in a data center."
Taking Care of the Applications
The FMS SCADA System is a monitoring system for environmental conditions and power in the Data Centers in addition to supporting equipment. Specific responsibilities are to support, develop, and maintain application and integration services. The System Integration team also oversees other SCADA systems that monitor and displays consolidated real-time process control data including power generation (Hydroelectric and Power Plants), gas storage facility, Hydrological/Meteorological data and various water systems and Irrigation Plant activity which is part of an overall effort to attain LCRA's business objectives. "Our goal at a high level, "we keep the SCADA systems running and current in addition to performing any development that is required," said Mike.
Again it is worth mentioning that by using Wonderware and its DA servers along with Kepware's iSNMP driver, we are able to utilize a single software package to monitor all aspects of the system. No longer do we need to have a separate program to monitor the SNMP devices. Report creation is accomplished within WonderWare also. Excel reports can be created on the fly or pre-programmed from any data that is collected and stored in the SQL database.
The LCRA System Integration group perceives a continuation of its relationship with Kepware with not only the iSNMP driver, but also with the software to build a redundant system. "Kepware has a redundancy module that allows us to swap over to another instance of Kepware that's running on a backup server. That program is in the works right now. Kepware has many other drivers we may use in the future as we continue to expand the system in support of LCRA business goals," Mike concluded.
Return to Kepware Case Studies Page »» |