Friday, December 5, 2014

Streaming Real-Time SCADA Data into Splunk

Posted by Erik Dellinger

If you are familiar with Splunk, then your first thought may be that they are an Operational Intelligence company gathering Big Data generated from log files within a data center. While this is true, it does not paint the entire picture of Splunk’s efforts around Big Data. Splunk has recently partnered with Kepware to harness data from the Industrial Internet of Things (IIoT). This blog will discuss some possible benefits of industrial machine data as part of business and operational intelligence strategy. It will also demonstrate the practical applications of the new partnership between Kepware and Splunk.

Talking "Big SCADA" at .conf2014

I recently attended the 5th Annual Splunk WorldwideAron Semle, Erik Dellinger, and Ray Labbe at .conf2014. Users’ Conference (.conf2014). The event attracted a lot of interest around industrial machine data. The people I spoke with were not your traditional industrial controls engineers who wanted to link some device that supports the Modbus protocol to some other device that supports the DNP3 protocol. Instead, these enthusiastic Splunk users were not very familiar with the protocols supported by most of the industrial controls market. For example, out of the few hundred people we spoke with, I can recall only two attendees who knew what OPC was. Instead of talking about OPC or Modbus, we spoke about Supervisory Control and Data Acquisition (SCADA) data in general terms—or as one presenter in an Oil & Gas Internet of Things informational session who referred to industrial machine data as “Big SCADA”.

The conference confirmed that the IT industry is truly interested in SCADA data. One of the challenges they will face is understanding what kind of data they can collect from an Industrial Control System (ICS) and what should they do with it. To add to this challenge, ICS are rarely, if ever, configured in the same way. They can be composed of thousands of different device types and software applications from any number of vendors. This is partly why the Kepware and Splunk partnership came to be. It is well-known that Kepware is the leader in providing connectivity to thousands of different types of devices for traditional SCADA purposes.

Differentiating Splunk from ICS Applications

How is Splunk different from traditional ICS applications like HMI, Historians, or MES and what is the purpose of SCADA data within a Big Data application like Splunk? HMI, Historians, and MES will continue to serve their much-needed purpose, but applications like Splunk are looking for direct access to important operations data that can be utilized for a wide variety of business and operational intelligence purposes. It is expected that the use case for considering industrial machine data in business and operational intelligence systems is going to be unique for every implementation, but we can look at some general applications.

For example, when we look at the manufacturing process of an automobile, there is a vast amount of data that could be collected for advanced analytics. Automotive companies are competing to manufacture safe, reliable, high performing, and fuel-efficient vehicles. And because the manufacturers of these vehicles are increasingly improving their vehicles in both quality and features, there is added complexity and a need for big data analytics.

To ensure quality, manufacturers are measuring the height, width, depth, and diameter of the parts produced that make up a component within an automobile. Looking at each individual component to ensure it is within the standards set forth by the manufacturer helps to ensure the engine will run properly when completed. Manufacturers also trace these measured components to look for outliers and know when to calibrate the equipment manufacturing the parts.

Using Big Data with applications like Splunk can take this a step further by looking not only at the measurement of single or multiple components over time, but also at data from different sources to provide a complete view into the manufacturing process. For example, comparing automobile service records from a dealership to the metrological traceability records generated during the manufacturing process could allow manufacturers to understand where,  when, and what tools were used to produce the faulty components. They can then analyze the data even further by looking at similar real-time trends in the manufacturing process to set schedules for tool and equipment calibration.

Enabling Data Correlation

In order for this type of data correlation to take place, you need flexible solutions that can sift through massive amounts of data quickly and from a wide range of sources—including the ICS. Splunk offers this solution, and Kepware’s Industrial Data Forwarder (IDF) for Splunk Plug-In is a quick and easy way to make sure that Splunk has access to the industrial machine data needed to extend its view into SCADA data. Splunk and Kepware enable IT and Operations to converge, creating an opportunity for new types of business and operational intelligence methods that were not historically feasible before.

Learn More

Have questions on how you can integrate machine data into your business and operational intelligence strategy? Leave a comment in the space provided below, or contact a Kepware representative by emailing or by calling +1 207-775-1660 x208. 

Download the IDF for Splunk Plug-In for KEPServerEX