One evening, a process plant control room operator noticed a flood of alarms on the operator consoles. Moments later, equipment at the facility tripped. All the alarms were pointing to the bad data received from an equipment control panel. In addition, data related to this process equipment was lost on the operator consoles for a few seconds. The control room operator immediately alerts the control systems engineer who takes a look at the systems alarm and event log. The engineer informs the operations team that the trip occurred because of a failure of PLC’s primary CPU in the equipment control panel. The data loss and alarm generation occurred while the secondary communication unit was coming online. The trip occurred because the secondary unit was not able to maintain the current condition of the process. The engineer is asked to explain why this had happened when both the primary and secondary units were in operation. Why did the secondary unit not take over control of the process immediately?
Having a primary and secondary unit does not always mean that the switchover is bumpless. A bumpless switchover is one where the equipment under control (EUC) is not disturbed when the switchover takes place. The switchover behaviour depends on whether the two units have a dual redundant or stand-by redundant configuration.
Let’s start by looking at the concept of fault tolerance. Fault tolerance is the ability of a system or subsystem to continue to be able to operate in the event of a failure of one or more of its components. Redundancy is the duplication of critical components to increase the reliability and availability of a system. Reliability refers to the probability that a system will operate and meet the required performance criteria for its desired operating period. Availability refers to the percentage of time that a system remains operational and serves its intended function.
It should be noted that fault tolerance does not always mean that the system has redundancy. For example, fault tolerance can allow a system to continue to function at a reduced level of efficiency. With redundancy, fault tolerance is achieved by having a backup system that takes over in the event of a component failure.
Most process control systems and safety systems offer redundancy of critical components like the PLC CPUs and the power supply units as a means of achieving fault tolerance for the system. The two main types of redundancy found in process control systems are N modular redundancy and standby redundancy.
In standby redundant systems, there is a primary unit and a standby unit. The standby unit is not usually kept in sync with the primary unit, so it must reconcile its input and output signals on the takeover of the EUC. Since the units are not synchronised, the transfer is not bumpless, which means that the secondary unit may send control signals to the EUC, that are different to the last control signals that came from the primary unit, before it failed. There are two types of standby redundant systems, hot standby and cold standby.
In cold standby systems, the secondary unit is powered off. On failure of the primary unit, the standby unit will have to be powered up manually and brought to the last known configured state of the primary controller. This results in increased downtime when compared with a hot standby system, leading to reduced availability of the control equipment. The only benefit of this arrangement, when compared with a hot standby, is that the reliability of the standby unit is maintained since it is powered off, when not in use.
In hot standby systems, the secondary unit is powered up and can optionally monitor the status of the equipment under control, taking over on failure of the primary unit. The standby controller functions as the watchdog to monitor the status of the primary controller so that it knows when to take over control of the process equipment. This reduces the system downtime and improves availability when compared to a cold standby unit. However, to monitor the equipment under control additional logic blocks must be written or configured in the controller, as part of application software (not built-in) specific to the equipment being controlled, to ensure that values of program variables are transferred to the standby controller periodically. Therefore, hot standby controllers can take a few seconds to switch over, depending on the application size and IO arrangements, resulting in a switchover that is often not bumpless.
To achieve bumpless switchover, N modular redundancy, where N units are running in parallel is recommended. The commonly available N modular redundancy configurations available in the market from process automation vendors are Dual, Triple and Quadruple modular redundant controllers.
Dual Modular Redundancy (DMR) is parallel redundancy, where the two units are running in parallel. The units are functionally equivalent, meaning that either of them can control the equipment under control. The two units are highly synchronised and receive the same input information at the same time. Their output values are then compared and a voter decides which output values should be used. This model easily provides bumpless switchovers since the primary and secondary units are completely synchronised.
While dual redundant modules provide fault tolerance and error detection when there is a difference in the output values from the two units, it does not provide error correction. In other words, it cannot determine which unit’s output is correct if they both operating and one has a malfunction that does not completely shut it down. Most dual redundant controllers would be configured to make the secondary unit take over in such instances by default, assuming that fault is within the primary unit that is currently controlling the EUC.
Triple Modular Redundancy
Triple modular redundant (TMR) systems offer improved reliability over dual modular redundant systems because they employ three functionally equivalent units running in parallel with three identical voting units. These systems can tolerate one additional failure when compared to a DMR system. In addition, they can provide error correction since the voting circuit can determine which unit is in error when a two-to-one vote is observed. Once the faulty unit is identified, it can be excluded from the vote and the TMR system then operates like a DMR system.
A lot of TMR systems available today also use diversity in the hardware or software used for the three modules. For example, one unit may be FPGA-based while the other two are microcontroller-based, with all three units performing the same function. This design helps address “common mode failures” and makes these TMR systems highly reliable. Common mode failures are failures where the components in the redundant system fail at the same time, in the same way.
Did you know? – SAPO (short for Samočinný počítač, “automatic computer” ) was the first fault-tolerant computer. It had three parallel arithmetic logic units, which decided on the correct result by voting, an example of triple redundancy. It was a computer and operated in the years 1957–1960 in Výzku mný ústav matematických strojů, part of the Czechoslovak Academy of Sciences.
Quadruple Modular Redundancy
Quadruple modular redundancy (QMR) takes the TMR reliability one step further in terms of tackling common mode failures. The QMR technology, has two modules, each with two processors running in parallel verifying each other. The processors are completely verified and tested for correct functioning. Thus, redundancy is established by the two processors within each module with separation, isolation and operation from the other module. This arrangement offers improved tolerance to common mode failures. In addition, the two modules also communicate with each other establishing redundancy between the two modules, making it a quadruple redundant system with voting taking place at two levels. The system can tolerate 3 processor failures in this system.
The terms Dual, Triple and Quad are generally marketing terms used by vendors for safety systems in the process automation industry. The architecture of these products in terms of voting arrangements and diagnostics is to be examined to fully understand the redundancy offered by the product. Diagnostic capability is a key differentiator in the different redundant architectures available for process automation equipment.
Failure modes and voting arrangements
Dual Redundant system could have a 1oo2, 2oo2 or 1oo2D voting arrangement. 1oo2D arrangement is used for most DMR systems available on the market. The “D” in 1oo2D stands for inherent diagnostics.
Before we discuss the voting arrangements, we need to understand the types of failures for safety systems. The safety systems can fail safely or dangerously. We are mostly concerned with dangerous failures. Such failure result in the system being unable to perform its functions when demanded by the EUC, which can lead to unsafe conditions for the EUC and damage to the equipment or loss of production.
There are two types of dangerous failures for safety systems, dangerous detected and dangerous undetected failures. Diagnostic capability within the system helps identify the failures which can be classed as detected failures. Similarly, safe failure can also be detected or undetected. Diagnostics helps to detect dangerous failures and converts them into safe failures, thereby, reducing the number of undetected dangerous failures, when compared to a system without diagnostics. The ability to detect failures depends on the strength of self-tests and comparative diagnostics between the subsystems of the redundant system.
Dual redundancy, for a safety system CPU, utilises two independent main processors and thereby offers a low probability of failure on demand when compared with a simplex (single processor) system. In other words, the possibility of a dangerous failure is reduced since the failure of one processor in a dangerous mode, the other one is still able to perform its function and safeguard the equipment. This, however, depends on the voting arrangements for the dual redundant system.
Dual redundant systems with 1oo2 voting are configured to degrade as in a 2-0 configuration. A 2-0 configuration will result in a fail-safe state of the safety system if either unit has a fault. This 2-0 degradation makes the safety system very safe for the EUC but offers low availability due to a high probability of spurious trips for the EUC. A spurious trip is a trip of the process due to a fault that is not triggered by a process condition but by a fault within the safety system.
The 1oo2D arrangement can help improve the availability of the system. When examining the 1oo2D arrangement, we can see that each unit monitors the other and the operating unit can shut down the failed unit. Spurious trips can be reduced with the 1oo2D arrangement since it degrades to a 1oo1D arrangement when a fault is detected in one of the units.
A dangerous failure of both processors can lead to an unsafe condition for 1oo2 and 2oo2 arrangements and the inability of the system to perform when there is a demand on the system. The 1oo2D arrangement fails dangerously only if both units fail dangerously and that dangerous failure is undetected. The number of dangerous undetected failures will be considerably lesser than a 1oo2 or 2oo2 system due to the diagnostics and the equipment can continue to be operated while the defective unit is being repaired.
Why consider TMR or QMR?
Considering what is offered by a 1oo2D arrangement, why do we need to consider a TMR or QMR system? One critical factor is the ability to continue to operate the EUC after a fault is detected. When used for safety-critical applications, the safety requirements place severe restrictions on the operating time after the 1oo2D system degrades to a 1oo1D mode of operation and the system is to be closely monitored during this period.
The TMR systems sold in the market typically use a triple module redundant architecture with diagnostics (2oo3D). The system will fail dangerously only if two units fail dangerously. On a detected failure of one unit, this system degrades to a 1oo2D for a 2oo3D system and then to a fail-safe after 2 units have failed. This is often depicted as 3-2-0.
It is interesting to note that a 2oo3 system (without diagnostics) will degrade to a 2oo2 arrangement for a dangerous failure of a unit and a 1oo2 arrangement for a safe failure of a unit. With the diagnostic coverage of a 2oo3D arrangement, the dangerous failures are detected and converted to a safe failure, hence the degradation to 1oo2D.
Examples of TMR products sold in the process automation industry are Rockwell Automation’s Trusted TMR safety system and Schneider Electric’s Triconex safety system.
Now on to QMR systems. Why consider them? Some articles indicate that TMR systems are not recommended to be operated with only a single unit because the internal diagnostics are not comprehensive to detect dangerous failures. Therefore, even for TMR systems, there are safety requirements that impose time limitations on operating a TMR system after the first unit has failed.
Quadruple modular redundant (QMR) architecture is based on 2oo4 or 2oo4D, with redundancy between the two modules and dual-processor technology in each module. The QMR architecture degrades in a 4-2-0 manner, however, after the first fault, the 1oo2D arrangement completely retains its internal diagnostics due to each module containing 2 processing units. This allows the system to operate in the 1oo2D arrangement without any time restriction. This ability to continue to operate without any repair time restrictions on the detection of the first fault gives it a significant advantage over the DMR and TMR systems.
Examples of QMR products sold in the process automation industry are HIMA’s HIQuad safety system and Honeywell’s Fail Safe Controller (FSC) safety system.
Adding redundancy increases the costs due to the additional product development time for the hardware and software. The cost increase can be up to 2 times and 3 times for Dual and Triple modular redundancy respectively. Whether your facility requires a DMR, TMR or QMR system will depend on the required safety criticality defined by safety integrity level (SIL) requirements and the production criticality defined by the cost of production loss at the facility due to downtime of the controller.
The cost of maintaining the systems should not be overlooked either. If spurious trips and performing maintenance repairs with limited time is a high concern for the facility, then the additional investment in a system with better reliability and availability can be easily justified.
To sum up, this article hopes to provide the reader with a high-level overview of redundancy and the different redundant safety systems available in the process automation sector. In-depth details of the architectures, failure modes and diagnostic coverage are not always easy to understand and each safety system available on the market has its unique features and capabilities. The process automation vendors and system integrators will be able to provide details on the redundant configurations and operating modes, including data on safety integrity, availability and product certification to industry standards like IEC 61508 and IEC 61511 so that end-users can make a more informed decision on the product that is right for their operating facility.
Related Articles
Further Reading
- Redundant Systems Basic Concepts, White Paper, National Instruments.
- The New Quad Architecture: Explanation and Evaluation, Dr. Lawrence V. Beckman, HIMA-Americas, Inc.
- Alarm Trip 1-Out-Of-2 Voting With High Availability, Moore Industries Worldwide