Many of the Industrial automation and control systems operating today are built using the ISA 95 standard – the international standard for the integration of enterprise and control systems. The ISA 95 model consists of 5 levels and is based on the Purdue Reference Model for Computer Integrated Manufacturing.
The process data of the operating plant is handled within levels 0 – 3. Systems at these 4 levels can be considered as associated with Operational Technology for the facility and are referred to as “OT” systems. OT systems are used to monitor and control the physical assets of an operating plant. Most of the industrial automation vendors like ABB, Emerson, Honeywell, Siemens, Yokogawa and others offer products that cover the equipment at all the OT levels.
Level 4 is where the data obtained from Levels 0 – 3 is shared with the supporting business functions to make decisions about forecasting future production and operational activities. Most of these systems at level 4 are related to Information Technology and are referred to as “IT” systems.
The Industrial Internet of Things (IIoT) and the Cloud
Historically, Supervisory Control and Data Acquisition (SCADA) systems had limited connectivity, with data from sensors available for analysis via the PLC only if the associated signals were configured within the SCADA. The data received was displayed on Human-Machine Interface (HMI) display screens or panels. Sensor data was collected and used primarily for supervision and control of the production process at Level 2 and was not easily available across a larger network that extended to Level 4.
The Distributed Control Systems (DCS) improved on SCADA systems by using a common database that linked the PLC or DCS controller data to HMI elements like faceplates that were pre-built and tested using object-based software libraries. DCS solutions also included historians that could store data from sensors as time series data for trend analysis. However, most of the OT data remained within Level 2.
A report titled “Unlocking the potential of the Internet of Things” [1] by McKinsey & Company highlighted that most data generated by existing IoT sensors are ignored by traditional systems. According to the report, in the oil-drilling industry, only 1% of the data from the 30,000 sensors on a typical oil rig is used and even this small fraction of data is used mostly to detect and control anomalies and not used for optimization, prediction or data-driven decision making.
The widespread usage of the Internet and the rise of Industry 4.0 technologies like cloud computing, big data and analytics and the Internet of Things (IoT), enable asset data to be analysed to improve asset management capabilities rather than just supervision and control of the production process.
Industrial IoT devices refer to a network of connected physical assets such as pumps, motors, relays, sensors and other field devices or “things” at Level 0 that perform a task in the physical world. The development and deployment of IIoT devices are leading to the convergence of IT and OT in an industrial setting. IIoT technologies allow devices to be networked so that data collection, transfer and analysis are seamless and sensor historic data no longer needs to be stockpiled at lower OT levels. The ability to make the sensor data available over a large network via IIoT, enables big data and analytics solutions like machine learning to perform an in-depth analysis of this data to gain operational insights and help operating companies make better business decisions.
With increased connectivity and a network of IIoT devices connected together, large amounts of data became available for analysis using complex compute-intensive algorithms and it became more feasible for companies to use cloud services to perform data analytics and machine learning tasks. Cloud services have already become a popular solution for Level 4 functions as they enable companies to focus on their core businesses instead of buying, owning, and maintaining physical data centres and servers.
IT systems are maintained by the cloud service provider, who offer different service models like Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS). Each of these service models offers companies the option to choose the level the flexibility and control they wanted in terms of the utilization of computing resources and storage. The cloud also offered rapid deployment and scaling of services based on the changing needs of customers.
The Digital Twin and Predictive Maintenance
Cloud service providers began to offer solutions where companies could use the IIoT-generated data to operate a digital twin of their facility in the cloud. A digital twin is a virtual model of a physical system designed to accurately reflect the physical system within a computer. The digital twin differs from a computer simulation because the data required for a digital twin is obtained from the real-world measurements obtained from the physical system.
To accurately reflect the physical system, large amounts of operating data must be acquired from a network of sensors connected to the physical system. For instance, a digital twin of a gas turbine would be a model or digital copy of the gas turbine built on a computer that would receive data from the various sensors measuring the pressures, temperatures and flows of the actual gas turbine operating at the facility.
The sensor data is used to understand the past and present operation of the physical system and to continuously improve the model of the digital twin with every second of operational data obtained. One can also imagine a scenario where data available from several assets can be used to improve the model of the digital twin, similar to how driving data received from cars is being used to improve the self-driving machine learning models for cars.
Once sufficient data is available, the digital twin can be used to run simulations of different operating scenarios in the virtual environment. These simulations can be used as lessons that lead to a better understanding of the performance and operational capabilities of the physical system. In addition, the data obtained from the sensors used in the condition monitoring of critical equipment, for example, vibration and speed data for a turbine, can be used within the digital twin to predict failures and implement predictive maintenance recommendations based on actual and real-time levels of stress and wear in the physical system. Artificial Intelligence (AI) techniques in the cloud are also allowing digital twins to learn from the vast amounts of process data and generate correlations that allow for optimization of process operating conditions which can, in turn, lead to better utilization and increased reliability of equipment.
The cloud also allows the OT personnel to access operational data remotely, which helps with making business decisions related to the optimization of maintenance activities through predictive maintenance leading to improved management of inventory and advanced planning of work for any call-outs to support the facility. OEMs can also provide remote support to investigate problems with the equipment based on the data collected in the cloud.
Limitations of the Cloud for IIoT
The benefits of cloud-based platforms were being realised by several businesses and there was a drive to converge IT and OT systems to have a unified environment with a single source of truth data from the Level 0 sensor equipment.
However, having a centralized solution in the cloud introduces latency issues since large volumes of data have to be sent and analysed at a remote location. Latency in simple networking terms can be defined as the total round trip time it takes for a data packet to travel from the source to the destination. The latency requirements for the various ISA-95 layers are shown in Figure [1]. To take full advantage of the digital twin and avoid missing critical opportunities to minimise energy loss or product waste, latency issues had to be addressed.
In addition, there are differences in cybersecurity priorities for IT and OT technologies. The IT teams prioritize Confidentiality and Integrity over Availability, while the OT teams prioritize Availability and Integrity over Confidentiality. OT systems focus on safe operations, production efficiency and minimising downtime. Ensuring OT systems are fault-tolerant with high reliability is crucial as system failures can result in loss of product, damage to equipment and even loss of life.
Lastly, security and data compliance requirements for production facilities located in certain geographical zones may prevent them from using cloud services that are available in different geographical zones.
These challenges of low latency, high availability and data sensitivity can be addressed with Fog and Edge computing-based solutions.
Fog Computing
Let’s discuss Fog computing before we move to the Edge. The literature on Fog computing seems to vary from the different sources reviewed and there is a lack of consensus on a standard fog computing solution. In fact, many commercial solution providers refer only to the cloud and the edge in their IoT solution architectures. A common interpretation of the Fog is describing it as a scaled-down version of the Cloud that is closer to the Edge and serves as an IoT gateway or Fog network node between the Edge and the Cloud.
One of the main applications of the Fog layer is to filter through the data received from the IoT devices and send only the relevant and non-sensitive data to the cloud. This improves the efficiency of the data transfer, reducing latency and allowing the cloud to directly use the data received without any need for data cleansing. In addition, network bandwidth and the amount of data storage space required in the cloud are also reduced, minimizing the costs associated with cloud services.
Some Fog computing solutions also recommend moving data analytics into the Fog layer, as the proximity of the Fog servers to the IoT devices, provides the improved network efficiency often required for real-time IoT applications.
Another argument in favour of using a Fog computing layer is the reduction of network congestion at the Edge due to the large amounts of data generated by IoT devices. The Fog layer can act as a data broker that handles large data volumes from IoT devices at the Edge, filters the data, minimises latency issues for real-time data analytics and takes away the network bandwidth requirements from both the Edge and the Cloud.
Researchers and enthusiasts of fog computing continue to investigate use cases for fog computing architectures. The research paper titled “A Cloud-Fog Computing Architecture for Real-Time Digital Twins” [2] showed that the average time per message received from field devices was reduced by more than 50% when fog computing was used for data handling before sending data to the cloud.
Despite the benefits offered by fog computing, practitioners are faced with challenges in the implementation and performance of fog based IoT systems mostly because there is insufficient real-world evidence to justify their use. Any experiments related to fog computing performance cannot easily be generalized to cover various scenarios that are present due to the vast variety of IoT platforms and devices that use different network architectures and communication protocols.
For many small or medium-sized applications, a direct IIoT Edge to Cloud architecture may be sufficient. In fact, for certain applications, the Fog may not be an ideal solution and the cost of the additional Fog layer between the Edge and the Cloud layer may outweigh the benefits.
With Fog computing being located on-premise, additional investment is required to integrate the systems in the Fog with the existing infrastructure of the operating facility, which can be a complex process. The Fog equipment will also need to be maintained by the operating facility and the flexibility, scalability and infrastructure management services offered by the cloud are lost. It is also more cost-effective to run data-intensive complex algorithms using the infrastructure offered by the cloud instead of the Fog if the real-time requirements are not too onerous.
Therefore, the deployment of a Fog computing layer at a facility needs a careful evaluation of the current system architecture, the number of IoT devices connected to the edge, the complexity of real-time data analytics required and the capabilities of the Edge computing layer.
Intelligent Edge Computing
This brings us to the Intelligent Edge computing layer. Similar to the Fog layer, the intelligent edge computing solution also attempts to collect, analyze, and process data from Level 0 more efficiently while addressing high compute costs, network bandwidth and latency issues that limit the feasibility of cloud computing solutions for many real-time industrial applications.
However, while Fog computing enables analytics to occur close to the edge in a Fog node or IoT gateway, intelligent edge computing pushes the algorithms and data processing functions directly into the embedded controllers called Edge programmable industrial controllers(EPICs), which are directly connected to the physical devices.
It is necessary to highlight the difference between edge devices and intelligent edge devices. Legacy architectures end up sending large volumes of data to the cloud as they rely on obtaining data from a non-intelligent edge device by polling the device at regular intervals, regardless of whether or not the data values have changed in the field. Intelligent edge devices like EPICs, on the other hand, have computing resources that pre-process data at the edge and only send relevant information to the cloud. Hence, these intelligent edge devices can analyze time-sensitive process data and return insights quickly allowing the necessary actions to be taken immediately before the opportunity is lost.
Most edge devices complement the control system and SCADA and do not replace them. The edge devices enable wider access to data and analytics from the field equipment, while the control systems control the field equipment. The edge controllers, on the other hand, have architectures that allow them to execute control logic similar to any PLC while including the additional computing power that runs in parallel to collect, process and analyze the data received from the physical devices. A few popular edge controllers available today are the groove EPIC controller from Opto 22, ControlEdge PLC controller from Honeywell and PACSystems from Emerson.
Edge computing also allows for direct connection to the Cloud, reducing the cost and effort required to deploy an IIoT solution by reducing the complexity of the IIoT architecture. It also improves communication reliability by reducing the number of components in the data path and hence the potential points of failure. By filtering and performing data analytics at the edge, the dependence on the cloud is reduced, the security of critical data can be maintained at the edge and cloud computing costs for data transmission and storage can be reduced by sending only relevant data to the cloud.
Shift in the ISA-95 model due to Industry 4.0 technologies
So where does this leave us when we try to relate the new Industry 4.0 developments back to the ISA-95 model? Several organizations have already migrated a large portion of their IT services to the cloud. However, the OT systems have traditionally remained separate from IT systems and organizations have relied on oversights from their business staff to collect data from the operational teams in order to make decisions on sales and inventory. The hesitancy to integrate IT and OT is because of the complexity of the architectures, especially when legacy systems with restricted communication protocols are present in the OT network.
Recent advances in IIoT, edge computing and big data analytics are now converging the worlds of IT and OT and allowing companies to harness their data directly from the physical devices to improve equipment and power use, optimize inspection and maintenance activities and reduce inventory costs.
There are efforts within the industry to understand how IIoT, edge, fog and cloud computing impact the Purdue and ISA-95 models. The European Union Agency for Cybersecurity (ENISA) has proposed a revised version of the Purdue model [3], that recognizes a Level 3-based Industrial IoT (IIoT) platform, which communicates directly with Level 1 IIoT devices.
Developments at the Edge
A lot of IIoT solutions today involve using EPICs to perform analytics at the edge and retain the asset data at the edge without sending any data to the cloud. This would be a preferable solution for many organizations that would prefer not to send data to the cloud for data security and privacy reasons. An edge-only solution may also be feasible for a large organization that has already invested heavily in on-premise computing infrastructure. For such organizations, a modification to the ISA-95 model involves the inclusion of IT systems at the edge in order to host critical applications within Levels 1 and 2, while non-critical applications that do not require real-time data processing are handled within the Cloud.
With the cost and power requirements of semiconductors decreasing and machine learning algorithms becoming more efficient, it is becoming more feasible to have solutions with AI and ML capability on embedded controllers. Microprocessor technology is continuously improving and a new class of chips called “inference chips” are now being designed that can bring AI and ML capabilities right into the edge devices. These are being developed by several manufacturers like Hailo’s 8 AI processor, Nvidia’s Tesla T4, Intel’s Myriad-X and Google’s Edge TPU. These advances in computing technology will further favour intelligent edge computing solutions over cloud-based solutions for performing machine learning and data analytics within a closed-loop system.
To facilitate data integration from multiple devices from different OEMs and to solve the challenges of adding new IIoT devices into existing industrial control systems, most of the modern, intelligent edge computing systems employ an open architecture that leverages standardized protocols like OPC UA and MQTT which increase vendor interoperability and help in reducing integration costs.
Both OPC UA and MQTT are IT-oriented and developed with built-in security. Such open protocols are also used to collect data from edge devices and transfer them to the cloud. OPC UA (Unified Architecture), which has evolved from the OPC (Object Linking and Embedding for Process Control) specification, is a client-server based architecture that is widely adopted by vendors and is one of the most preferred way to share data between devices from different vendors.
MQTT (Message Queueing Telemetry Transport) is a publish-subscribe model where a central server acts as a broker managing data delivery for the entire network. MQTT enabled devices to publish data only when a monitored tag changes value. The data can be accessed by any device that subscribes to the data via the MQTT broker instead of a point-to-point connection. This MQTT communications model delivers an 80% – 90% reduction in bandwidth consumption compared to continuous poll-response point-to-point models. An interesting article comparing these two protocols has been published by the company HiveMQ.
To solve the problem of slightly different data naming conventions for asset data tags by different vendors of the same type of equipment, like data tags for a motor, for example, another interesting development at the edge is the provision of a unified namespace. Companies like Highbyte, offer a unified namespace solution that acts as a centralized repository of all the IoT data, allowing clients to use OPC UA or MQTT to readily access the data from field devices. The unified namespace collects data from various sources in their different naming conventions, contextualizes it and transforms it into standard formats that all the systems connected to the namespace understand. These unified namespaces become increasingly relevant when organizations want to scale their IoT solutions and connect several networks at the edge.
The industrial facility owners will ultimately have to decide which edge solution they would like to adopt. Some customers may prefer an edge platform that is hardware agnostic which offers more flexibility for future expansion and some may prefer one that locks them into buying edge controllers from the same vendor providing the edge software platform which offers ease of maintenance in the long run.
The role of the Cloud
Information widely available suggests that companies will attempt to standardize the edge controllers and IIoT platform solutions for their facilities. However, companies will continue to send at least some data not required for real-time analysis from the edge to a cloud taking advantage of the scalability and flexibility that are offered by infrastructure, platform and software as a service models offered by the cloud.
Large operating companies that have multiple facilities across the globe can consider having both edge and cloud architectures, where the cloud capabilities are used to share and analyze data from the different sites, allowing the entire organization to benefit from the data insights that are managed centrally.
There is a very interesting case study [4] of how the biotech company Moderna, has used the Amazon Web Services (AWS) cloud platform for the operation of their pharmaceutical manufacturing facilities. Moderna makes use of many of the AWS cloud services from the 200+ services on offer by AWS.
Given the large number of SaaS applications available in the cloud, companies want to be able to leverage these services while also retaining control of the infrastructure and not depend on cloud services that needed to be accessed remotely via an internet service provider. This led to the development of on-premise cloud architectures where cloud applications required are installed on a company’s own servers and behind its firewall making them reliable and secure. Cloud solution providers now offer new hybrid cloud architectures where a private cloud is constructed either on-premise or through a hosted private cloud provider and is connected via a WAN to a public cloud platform from a trusted third-party provider for additional non-critical cloud services.
IT vendors are also prioritizing scalability and flexibility by offering IIoT platforms such as PTC’s ThingWorx, OSIsoft’s PI System, Microsoft’s Azure IoT and Amazon’s AWS IoT. OT vendors are also offering edge platforms and partnering with cloud service providers, for example, ABB’s Ability and Schneider Electric’s Ecostructure digital platforms operate with Microsoft Azure. Inductive Automation’s Ignition and Siemens Mindsphere can be used with either Amazon AWS or Microsoft Azure.
Digitalization has become an important factor for companies to stay competitive and the space between the edge, fog and cloud is evolving and will continue to do so as organizations gain experience with more IIoT deployment case studies.
Further advances in Industry 4.0 technologies like AI and ML algorithms for big data, edge and cloud computing capabilities and the application of emerging cybersecurity frameworks will play a crucial role in the development of a widely accepted model for the integration of enterprise and control systems that incorporates these technologies.
References and Further Reading
- Unlocking the potential of the Internet of Things by McKinsey Global Institute, June 2015.
- A Cloud-Fog Computing Architecture for Real-Time Digital Twins by Francisco P. Knebel*, Juliano A. Wickboldt and Edison P. de Freitas, Institute of Informatics, Federal University of Rio Grande do Sul, Porto Alegre, Brazil.
- Good Practices for Security of Internet of Things in the context of Smart Manufacturing – The European Union Agency for Cybersecurity (ENISA).
- Modernizing pharmaceutical manufacturing with IoT and AI/ML (MFG203) by Patrick Buckner, AWS and Roland Smith, Moderna – (YouTube video link).
- IoT edge computing – what it is and how it is becoming more intelligent by Matthew Wopata, IoT Analytics.
- Advancing from industrial automation to industrial autonomous operations by Mark T. Hoske, Control Engineering.
- What is IT/OT convergence? Everything you need to know by Stephen J. Bigelow and Ben Lutkevich, TechTarget.
- Unified Namespace: The Hub and Spoke Model for Your Data by Will Aja, Panacea Technologies.