Lessons

Redundancy is a fundamental design principle in real-world systems: it involves including additional components, subsystems, or resources beyond those strictly necessary to perform a task. These additions can either remain inactive until a primary component fails—cold redundancy—or operate alongside the primary so that handover is immediate—hot redundancy. Three distinct forms arise in practice: structural redundancy duplicates physical components (motors, processors, sensors); functional redundancy provides different subsystems capable of fulfilling the same function via varied mechanisms; and informational redundancy adds extra data or signals to support error detection and cross-checking.

Redundancy is most effective when component failures are statistically independent. In practice, however, many systems face correlated failure modes that undermine this assumption. Redundant transformers may fail simultaneously during a heat event; redundant sensors may all be affected by the same electromagnetic interference; processors from the same manufacturing batch may degrade at the same rate under the same thermal stress. Diversity—using components from different vendors, routing communication over physically separate paths, or implementing diverse software architectures—reduces the likelihood of simultaneous failure and maintains the intended protective value of redundancy.

Sensor redundancy is frequently combined with majority voting: if three sensors measure the same physical variable and one produces a deviant reading, the system discards the anomalous input and continues using the remaining two. Figure 3.1 illustrates eight real-world implementations. For reinforcement learning deployed in real-world settings, redundancy corresponds to the ability to maintain a functioning policy under partial component failure—a principle directly embodied in the distributed execution model of Chapter 5.

Sensor fusion combines data from multiple sensors to produce a unified and more reliable representation of the environment. It is essential in systems that must operate under uncertainty, noise, or partial observability. Several well-established algorithms underpin practical implementations: Kalman filters and their nonlinear variants (EKF, UKF) provide mathematically grounded methods for continuous state estimation; Bayesian approaches offer a broader probabilistic framework; and particle filters or data-driven models are used where precise modelling is difficult or computational efficiency is paramount.

Autonomous vehicles illustrate the complementarity that motivates sensor fusion: cameras capture rich visual information for object classification; LiDAR provides high-resolution 3D spatial data; radar performs reliably in adverse weather; and GPS supports global localisation. Manufacturers such as Waymo integrate all four modalities into their core architecture, achieving more consistent and resilient performance than any single sensor could provide. In robotics, Boston Dynamics platforms fuse IMU data with stereo vision and force feedback to achieve dynamic locomotion over uneven terrain. Figure 3.2 illustrates how these input modalities combine across application domains.

Sensor fusion can be centralised—raw data transmitted to a single processing unit for joint optimisation—or decentralised, with local nodes performing preliminary fusion before passing results to a coordinator. Centralised architectures allow tighter integration but demand significant bandwidth; decentralised designs reduce communication overhead and support fault tolerance at the cost of synchronisation complexity. For reinforcement learning, sensor fusion corresponds to partial observability management: the observation function \(\mathcal{O}\) aggregates heterogeneous inputs before the policy acts, and the architecture of that aggregation affects both accuracy and latency.

A failsafe is a design feature intended to move a system into a safe state when a fault is detected. Unlike redundancy, which attempts to maintain normal operation despite component failures, a failsafe prioritises safety over continued function. Two archetypal responses illustrate the range: a robotic arm that halts immediately when it detects unexpected resistance, and an aircraft that transfers control to the pilot when automation encounters a fault it cannot resolve. In both cases, the system acts to bound harm rather than to preserve performance.

Failsafes are activated by cross-checks: multiple independent sensors or monitors are consulted and compared. If two navigation systems report conflicting positions, the autopilot may disengage and alert the flight crew. Triggers may also be timeouts (a process fails to complete within an expected window), out-of-range readings, or loss of communication. When one of three sensors produces a deviant reading, a majority voting scheme can continue operation using the remaining two; when disagreement is severe or irresolvable, the system transitions to the safe state. In nuclear power plants, that state is unambiguous: control rods drop into the reactor core automatically, shutting down the reaction without waiting for human intervention. The “safe state'' is always defined relative to the application's risk profile—shutdown for some systems, a conservative limited-function mode for others.

Failsafes are implemented in hardware, software, or both. Hardware failsafes—pressure-relief valves, mechanical interlocks—address physical failure modes independently of software. Software failsafes respond to a broader range of conditions but are themselves susceptible to bugs; “watchdog'' hardware addresses this by monitoring software execution and enforcing a reset if anomalies are detected. For reinforcement learning, failsafe semantics correspond to safety-constrained terminal states: the policy must be designed to reach or respect them, not to optimise through them.

As Adam Smith's pin factory illustrated, modern systems achieve efficiency through distributed specialisation: components are produced or processed separately and brought together only at the point of integration. Within a single autonomous vehicle, multiple subsystems—perception, navigation, control, communication—operate in parallel on separate hardware units, each optimised for its own task. Coordination across these units demands robust protocols for synchronisation and fault management, but the separation itself is what makes specialisation possible.

Hierarchy is the organisational principle that makes large-scale distribution manageable. In industrial process control, low-level programmable logic controllers (PLCs) manage rapid, time-sensitive operations—valve control, temperature regulation—whilst supervisory control and data acquisition (SCADA) systems handle high-level monitoring and decision-making across entire facilities. Air traffic management extends this to three levels: local aircraft systems manage immediate flight dynamics; pilots handle tactical situational awareness; centralised ATC coordinates regional and national airspace. Each level operates quasi-independently, which means failures at one level are isolated and do not cascade automatically into the others.

Distribution enhances transparency and resilience because defective subsystems can be examined in relative isolation. Spreading functionality across multiple independent units makes faults easier to diagnose, facilitates targeted maintenance, and supports natural redundancy through overlapping roles. For reinforcement learning in real-world settings, this hierarchy of timescales and responsibilities directly motivates the modular policy architecture examined in later chapters. The case studies that follow show how these general principles are realised under much tighter operational and safety constraints.

Protections are mechanisms that enforce operational boundaries and prevent unsafe actions. They constrain system behaviour to remain within safe or allowable parameters, acting as safeguards against misuse, malfunction, or unforeseen environmental influences.

In aviation, flight envelope protections are the defining example. Fly-by-wire aircraft include control laws that prevent the aircraft from exceeding critical aerodynamic or structural limits—excessive pitch, bank angle, airspeed, angle of attack, or load factor. In an Airbus aircraft operating under normal law, the system actively limits control inputs that could induce a stall or over-G condition. Such protections reduce pilot workload by embedding aerodynamic constraints directly into the control interface; the pilot works within a pre-constrained envelope rather than manually avoiding its boundaries.

Energy systems rely on analogous layered schemes. In electrical grids, protective relays monitor voltage, current, and frequency to detect abnormal conditions such as short circuits or overloads; when triggered, they isolate the affected section to prevent wider instability. In nuclear power plants, automatic shutdown sequences engage if critical parameters exceed safe limits, ensuring the plant enters a safe state without requiring human intervention. For reinforcement learning, protections correspond to constrained action spaces and safety-critical terminal states—the policy graph architecture in Chapter 5 enforces analogous boundaries through hard routing and commitment bounds. The following section turns from the spatial constraints imposed by protections to the temporal constraints imposed by latency.

Latency is the time delay between an input or event and the corresponding system response. In remote operations—UAVs, telesurgical instruments, bomb disposal robots—delays of even a few hundred milliseconds can result in misalignment, errors, or loss of situational awareness, because operators rely on immediate visual, haptic, or auditory feedback to guide precision tasks. Low latency is often framed as the primary goal, but network congestion introduces jitter: variability in delay that disrupts the timing of control loops and is often more detrimental to performance than moderate, stable delay [27].

Recent evidence makes this concrete. Noguera Cundar (2023) [28] examined latency variability in a teleoperated ultrasound robot: under standard WLAN, latency fluctuated between 33 ms and 750 ms, producing a maximum positional error of 7.8 mm. Switching to an isolated VLAN reduced the average delay by approximately 200 ms and cut positional error by 70%, to 2.4 mm. Kelkkanen (2023) [29] found a complementary result in a VR aiming task: unpredictable target motion impaired performance at around 90 ms latency, whilst predictable motion extended the usable threshold to approximately 130 ms. Together, these studies demonstrate that consistent, predictable latency is more important for precise real-time control than achieving the lowest possible but highly variable delay.

Systems with critical latency requirements manage this through two complementary strategies. Dedicated links—leased lines or isolated VLANs—provide fixed bandwidth and temporal determinism at the cost of infrastructure overhead [30]. Local fallback mechanisms provide the alternative: equipping the local device with onboard autonomy to hold position, return to a safe waypoint, or follow a preprogrammed routine when real-time remote input is unavailable [31][32]. Systems establish clearly defined latency thresholds—derived from empirical testing or risk analysis—beyond which the device transitions to a fallback mode rather than continuing degraded remote operation.

Effective latency management therefore depends not only on minimising delay but on maintaining stability and predictability under diverse conditions. Well-managed latency allows users to form accurate mental models of system behaviour, enhancing both trust and task performance [33][34]. Distributed control and robust fallback mechanisms are the principal engineering responses, and both recur in the case studies that follow.

The Airbus A320 marked a turning point in civil aviation by introducing the first fully digital fly-by-wire (FBW) control system in a commercial airliner. Rather than mechanically linking the cockpit controls to the flight surfaces, the FBW architecture interposes digital flight control computers: pilot inputs are interpreted, filtered, and translated into surface commands according to the active flight law. This computer-mediated control reduces pilot workload, enables envelope protections, and supports structured degradation when faults occur. The A320 offers an unusually well-documented case study in how layered automation, distributed computation, and redundancy are engineered into a system that must remain safe across a wide range of failure conditions.

The A320 replaced traditional control columns with side-sticks, which transmit pilot inputs to the FBW system. The FBW system determines the desired aircraft state and issues appropriate commands to the flight control surfaces; rudder pedals similarly feed through digital flight control computers rather than direct mechanical linkage.

The A320 flight deck relies on a clear separation of roles between the flight crew and the onboard systems. Typically, two pilots operate the aircraft: the pilot flying, responsible for flight path control, and the pilot monitoring, who manages communication, systems monitoring, and checklist execution. This human division of labour is mirrored in the architecture of the computerised part of the A320, which distributes responsibilities across multiple functionally distinct computers. A broad overview of the different systems in the A320 is shown in Figure 3.4, along with the different forms of communication that exist between them. Elevator and Aileron Computers (ELACs) manage the aircraft's primary pitch and roll control, whilst Spoiler and Elevator Computers (SECs) serve as a backup for pitch and roll and control the spoilers. The Flight Control and Guidance Computers (FCGCs or FCCs) process autopilot, flight director, and auto-thrust commands. These separate systems each handle discrete but interdependent functions, ensuring that no single failure compromises overall control. The structured separation of roles—both human and computational—enhances system transparency, reduces operational workload, and improves the ability to isolate and manage failures.

Structural redundancy is critical to the safe operation of the A320. The aircraft does not depend on any single point of control or measurement. Instead, it uses multiple, independent pathways to perform critical functions. This principle extends across both hardware and information domains. The aircraft incorporates redundant sensors—such as multiple angle of attack vanes, pitot tubes, and inertial reference systems—which feed data into the flight control computers. The system compares these inputs to identify inconsistencies, known as `cross-checking'. When the data from one sensor differs significantly from the others, the system flags the anomaly to the operator and excludes the faulty input from critical calculations. This form of informational redundancy maintains system integrity even amidst degraded or failed components.

Flight control computers in the A320 also implement structural redundancy. The ELACs and SECs each operate in pairs and each FCC includes multiple processing lanes. If one processor or module fails, another can assume its function without disrupting flight or causing safety issues. The system performs a cross-check, monitoring the consistency of outputs across computers in real time. When a disagreement arises between redundant units, internal logic can determine the correct value, such as through voting or by excluding outliers. This approach allows the aircraft to isolate and contain errors before they affect downstream systems. These redundancies do not just act as backups; they form a distributed system for efficient fault detection, identification, and mitigation. The flight control system is illustrated in Figure 3.5.

Hydraulic systems, responsible for powering the flight control surfaces, landing gear, brakes, and thrust reversers also follow a similar redundancy model. The A320 includes three fully independent hydraulic circuits, colour-coded green, yellow, and blue. Each system draws power from different sources. Figure 3.6 shows how the source from which each system is initially derived and from which each can derive backup power. The green and yellow systems rely on engine-driven pumps, whilst the blue system is powered by an electric pump. In some scenarios, such as electrical failure, the blue system can also receive power from a Ram Air Turbine (RAT), which deploys into the airstream to generate hydraulic pressure. These three systems supply overlapping sets of actuators, ensuring that each critical control surface remains operable even if one or two hydraulic systems fail. The separation of hydraulic power sources prevents localised mechanical failure from propagating into system-wide control loss.

Engine redundancy is another important feature of modern aircraft. Although designed for optimal performance with both engines functioning, the A320 can safely complete a flight with only one operational engine. The remaining engine provides sufficient thrust for level flight and controlled descent, whilst key systems remain powered through electrical and hydraulic cross-connections. The FBW architecture adapts automatically to the new flight envelope, adjusting control laws to ensure safe handling. The functional redundancy of having two pilots also means that, if one pilot becomes incapacitated, the aircraft can continue to be flown safely. In this way, the A320 maintains functional integrity through layered redundancy, enabling safe and managed degradation rather than catastrophic failure.

In addition to hardware and sensor redundancy, the A320 employs a layered hierarchy of flight control modes known as flight laws. These laws—normal, alternate, direct, and mechanical backup—govern the behaviour of the fly-by-wire system under different operational conditions. Figure 3.7 summarises each mode and the conditions under which transitions between them occur. In normal law, the system enforces full flight envelope protections: pitch, load factor, bank angle, and stall prevention. If certain failures occur, the system reverts to alternate law, which preserves some protections whilst allowing wider pilot authority. Direct law removes all protections, translating side-stick inputs directly to control surface movements (Figure 3.8). In rare cases of complete electronic failure, mechanical backup allows direct control of the flight surfaces. This tiered approach ensures that pilot authority increases progressively as system assistance decreases, allowing continued control in a wide range of degraded scenarios.

Pilots operating the A320 are trained to prioritise aviate, navigate, communicate: safe flight first, then navigation, then communication. This priority ordering is embedded in procedure, not just advice, and reflects the same principle as the flight laws themselves: the most safety-critical function is preserved at every level of system degradation.

The Airbus A320 illustrates how layered architecture, distributed computation, and structured redundancy can be engineered into a system that manages complexity without sacrificing transparency or safety. Crucially, its design anticipates off-nominal conditions, incorporating mechanisms to absorb and adapt to them without loss of safe operation. Several principles emerge with broad relevance to automated systems. The separation of roles—between pilots, between software components, and across hardware layers—supports clarity of function and traceability in the event of anomalies. Redundancy, both informational and structural, ensures that no single failure in a sensor, computer, or control surface propagates into wider system failure. The tiered flight laws define constrained transitions between operational modes: each law specifies what protections are active, what the pilot controls mean, and under what conditions the system degrades to the next level. This is the direct structural precedent for the constrained transition semantics of policy graphs in Chapter 5.

The Kangduo surgical robot presents a different realisation of these principles, where the central challenge is not flight envelope protection but latency management across a network.

Operating increases a surgeon's hand tremor by approximately 8.4 times compared to desk work [35][36], and traditional open surgery forces prolonged awkward postures that increase musculoskeletal strain [37]. Patient travel for medical treatment costs Americans \(89bn per year~\cite{rhyan2019travel}, much of which reflects the need for specialist procedures to be concentrated at major centres. Robotic surgical systems—standardised by the da Vinci platform since the early 2000s—address these constraints by providing tremor filtration, motion scaling, and ergonomic consoles~\cite{IntuitiveDaVinci}. However, a typical da Vinci system costs \\)1.5--2 million, limiting adoption to well-resourced institutions. The Kangduo KD-SR-01, first registered by China's National Medical Products Administration in 2022, was designed as a substantially lower-cost alternative for the domestic Chinese healthcare market. It incorporates the core elements of master--slave robotic surgery—three-arm configuration, ergonomic open console, and high-definition 3D imaging—and is instructive for this thesis because of what its telesurgical deployments reveal about latency.

Clinical evaluations have reported acceptable safety and efficacy across prostatectomy, nephrectomy, and urological repair [40][41][42]. The evidence supports cautious claims about feasibility and surgeon ergonomics in the reported settings; the focus here is on the networking constraints that emerge when the system is extended to telesurgical use.

Published descriptions of the KD-SR-01 indicate that it combines tremor filtration with motion scaling to improve fine control. The available literature does not provide enough low-level implementation detail to justify a stronger claim about the precise filtering method, but it is clear that the system is designed to smooth surgeon input and support delicate manipulation.

The KD-SR-01 system is designed with an integrated dual-control interface that allows operative control to pass from the primary surgeon to a secondary surgeon. Figure 3.9 shows the relationship between different ways in which the system can be controlled. In this configuration, the primary surgeon initially manipulates the robotic instruments from the master console; during critical phases of the procedure, the system permits the secondary surgeon to assume control. This handover is presented as a synchronised transition intended to avoid disrupting the flow of the operation.

Robotic subsystems—including arms, actuators, and endoscopic tools—must continuously perform self-diagnostics to detect anomalies, initiate fail-safe modes, or prompt handover to manual control where necessary. Handover mechanisms themselves need to be secure and non-disruptive, allowing rapid transition of control authority without interrupting the surgical workflow.

The published material is more informative about operational features than about low-level implementation. The safest conclusions are therefore limited ones: tremor filtering, motion scaling, workspace constraints, and rapid handover are treated as safety-relevant features, but the available papers do not provide enough detail to support stronger claims about the internal power, filtering, or fieldbus design of the platform.

The KD-SR-01 system also depends on robust networking to facilitate real-time communication between its master console and the robotic arms. The reported studies use dedicated wired links or mixed 5G and wired configurations, which is enough to show reliance on stable low-latency communication, but not enough to specify the underlying real-time networking stack in stronger terms.

Latency management is extremely important. A recent clinical and animal study by Fan et al. (2023) [43] evaluated the feasibility of dual-console telesurgery using both wired and fifth-generation (5G) networks. The experimental networks for the wired and 5G trials are shown in Figure 3.10. In the animal model, partial nephrectomy was performed remotely over an 80 km distance via a dedicated Internet Leased Line, achieving a mean latency of 130 ms (range 60--200 ms) and a control swap time of just 3 seconds, with no observed complications or packet loss exceeding 1%. In a subsequent human trial, a 32-year-old patient underwent remote pyeloplasty with a mixed 5G and wired network configuration. The mean latency reached 271 ms (range 206--307 ms), but performance remained within clinically acceptable limits in that setting. The study treated latency below 300 ms as workable for telesurgery, whilst also noting that lower values are preferable for longer procedures or critical surgical steps.

One of the primary technical challenges in telesurgical systems such as KD-SR-01 lies in ensuring stable, predictable latency. While average latency below 300 ms may be clinically acceptable, variability—known as jitter—can severely impact surgical precision and operator confidence. Even when nominal delay falls within thresholds, sudden spikes can desynchronise hand–eye coordination or disrupt visual feedback, particularly in procedures requiring fine motor control. As demonstrated in the dual-console clinical trial, latency values ranged from 206 ms to 307 ms, with a mean of 271 ms.

The KD-SR-01's dual-console design introduces an important layer of redundancy: should the remote surgeon experience network failure or degraded responsiveness, local control can resume within three seconds. This sub-three-second swap is relevant not only to training and oversight but to continuity of care under degraded communication conditions.

The published reports also make clear that latency is treated as an operational safety variable, even if they do not fully document the console-level monitoring interface. That is enough for the present argument: stable communication is part of the control problem rather than a background implementation detail.

Telesurgical systems like the Kangduo provide a useful example of automation operating within a real clinical service. The Kangduo acts as one node in a wider operational system involving a human surgeon and several other clinical systems in the operating theatre. The roles are clearly defined: the primary surgeon makes decisions and acts, the network communicates those decisions, the Kangduo interprets the command, applies tremor smoothing and safety checks, and sends high-quality 3D imaging back to the operator.

Experimental telesurgical deployments of the system have shown that whilst low latency remains desirable, consistent and stable latency is more important for operational safety and human performance. The dual console design and presence of a local fallback provide important redundancy. The required presence of a surgeon with the patient means that, even in the case of a network failure or a serious mechanical issue, a qualified human operator can intervene and prevent complications arising.

Réseau de Transport d’Électricité (RTE) is France’s independent transmission system operator, responsible for transmitting electricity from large-scale generation sources—nuclear, hydroelectric, wind, and solar—to regional distribution networks serving nearly 67 million residents. It also facilitates substantial cross-border exchanges, combining technical and economic management in a single operational remit.

The network encompasses approximately 100,000 kilometres of high-voltage transmission lines and produces roughly 490--500 TWh of electricity annually (recent years). Power is transported at 400 kV along main arteries to minimise resistive losses; approximately 2,200 transforming substations step it down for regional distribution and ultimately for consumption.

The system employs a large-scale distributed, hierarchical and redundant topology. Each substation uses so-called Intelligent Electronic Devices (IEDs). An IED is a microprocessor-based controller that integrates functions such as protection, control, monitoring and communication within the substation environment. These devices continuously measure electrical quantities—such as current, voltage and frequency—and execute protective relaying functions. For example, when an IED detects that parameters exceed predetermined thresholds, it can initiate actions like opening or closing circuit breakers to isolate faults, re-route power flows, or adjust transformer tap settings, thereby preserving the stability of the system. The inherent intelligence and rapid response time of IEDs are essential for handling unforeseen contingencies, reducing downtime and minimising the risk of widespread outages. IEDs allow for an element of distributed autonomy, where individual subsystems can make decisions to even out demand and respond to localised failures without the need for external intervention.

Local redundancy is implemented through the deployment of multiple IEDs, often on parallel communication channels within a substation. This ensures that if one device fails, another can assume the necessary control and protective functions without interruption. Additionally, local control systems in substations are designed to monitor a range of conditions—including overcurrent, undervoltage, or frequency deviations—and to respond autonomously. Actions taken at the local level may include the tripping of specific feeders, reconfiguration of network topology, and activation of backup generation or storage systems. These measures are typically executed in real time in response to transient faults, sustained overloads, or sudden changes in demand, contributing significantly to the overall robustness and fault tolerance of the grid.

IEDs detect common fault classes—overcurrent, high impedance, voltage drop, phase imbalance, and surge—and respond autonomously by isolating affected circuits, reconfiguring network topology, or adjusting transformer settings, without waiting for instruction from a central controller.

IEC 61850 defines the architecture, communication protocols, and data models for substation automation. It standardises high-speed data exchange between IEDs and control systems—using mechanisms such as GOOSE messaging and Sampled Values—enabling real-time communication within the substation and reducing engineering complexity through standardised configuration language (SCL) files.

The French system uses an industrial standard known as Supervisory Control and Data Acquisition (SCADA) for the monitoring and control of electrical systems. It provides a platform through which aggregated information from IEDs is communicated to control centres. SCADA systems receive data on various operational parameters, such as voltage levels, current flows, frequency, equipment status, fault indicators and alarm signals. These measurements, along with real-time diagnostics and status reports, are transmitted over redundant communication networks.

Redundancy is a critical feature of substations, ensuring that even if one component fails or communication paths are disrupted, alternative routes can maintain the flow of information. IEDs, often installed in structurally redundant configurations, continuously monitor the substation's operational parameters and are programmed to initiate protective measures by signalling faults or abnormal conditions. The SCADA system interprets these signals to trigger automated responses, such as isolating faulted segments or redistributing loads, whilst simultaneously alerting operators for manual intervention if necessary. Through a combination of hardware redundancy, dual communication channels and integrated fault-detection algorithms, the overall system is engineered to provide a reliable, resilient and timely response to both transient and permanent faults, preserving system stability and ensuring the continuous operation of the grid.

For this thesis, the RTE case contributes a different lesson from the A320 and Kangduo examples. It shows that real-world reliability is often achieved not by a single intelligent controller, but by layered autonomy: local devices handle faults quickly, regional structures coordinate wider responses, and central systems maintain system-wide balance. That pattern matters for reinforcement learning because it suggests that distributed control should be designed around clearly separated responsibilities, bounded communication roles, and explicit fallback structure rather than around a single monolithic policy. In that respect, the grid looks less like one optimiser and more like a managed collection of specialised units operating at different timescales. This pattern—local fast responses at the IED level, slower regional coordination, and global optimisation at SCADA—directly parallels the timescale decomposition in Chapter 5, where distributed policy units operate on commitment bounds that enforce analogous temporal separation.