Effects

In Chapter 3, we examined how real-world systems manage complexity through carefully engineered patterns of specialisation, hierarchy, and delegation. The Airbus A320 achieves reliable flight control by distributing responsibility across dedicated computers—Elevator and Aileron Computers (ELACs) for pitch and roll, Spoiler and Elevator Computers (SECs) for backup control, and Flight Control and Guidance Computers (FCGCs) for higher-level coordination—each operating within well-defined interfaces and constrained transition rules embodied in the aircraft's flight laws. The French power transmission network maintains grid stability through a three-tier hierarchy: local Intelligent Electronic Devices (IEDs) respond autonomously to immediate faults, regional substations coordinate load balancing, and a central control system (CNES) manages nationwide demand. The Kangduo surgical robot enables remote telesurgery by implementing explicit handover semantics between local and remote surgeons, with sub-three-second delegation transitions and robust fallback to local control when network conditions degrade. These systems share a common architecture: specialised units with distinct responsibilities, coordinated through explicit delegation and return mechanisms, operating under hard constraints that ensure predictable, accountable behaviour. These principles trace to Adam Smith's division of labour—the insight, elaborated in Chapter 2, that specialisation and coordination drive productivity.

This chapter proposes policy graphs as a formalism that distils the architectural patterns observed in Chapter 3 into an implementation-ready abstraction for modular reinforcement learning. A policy graph is a directed graph \(G=(V,E)\) whose nodes are callable policy units—analogous to the A320's flight computers or the power grid's IEDs—and whose edges constrain permissible delegations, much as the A320's flight laws govern transitions between control modes. Execution follows explicit call-and-return semantics: at any time a single unit is active, and it may (i) act in the environment, (ii) delegate control to a permitted successor, or (iii) return control to its caller. This mirrors the surgical robot's dual-console handover, where control authority transfers cleanly between operators with unambiguous responsibility at each moment.

Policy graphs address three gaps left by existing hierarchical RL formulations. First, they provide operational semantics that are directly implementable: delegation is a first-class operation with defined preconditions (edge constraints), commitment bounds prevent unstable switching (analogous to the A320's phase-based cockpit communication rules), and call traces provide accountability (as required for debugging real systems). Second, they unify diverse hierarchical patterns—options, feudal hierarchies, manager--worker systems—within a single framework whilst enabling non-tree topologies that better reflect real-world redundancy and shared subskills, much as the A320's three hydraulic circuits provide overlapping coverage of critical actuators. Third, they expose explicit control points for deployment constraints: individual units can be trained, tested, swapped, or distributed across heterogeneous hardware independently, whilst routing decisions remain inspectable and constrained, addressing the transparency and modularity requirements identified in real-world automation systems.

In the environments that motivate this work—web browsing, file-system interaction, and similarly long-horizon, interface-driven domains—complexity arises from the need to compose many precise, low-level operations into coherent workflows. Monolithic end-to-end policies struggle in this regime: credit assignment becomes difficult under sparse rewards, training is unstable when perception and control are learned jointly, and inference cost remains constant even when only a small subset of behaviour is relevant. Policy graphs address these challenges by enabling specialisation (low-level units master recurring interaction primitives), coordination (a router sequences units to achieve long-horizon objectives), and conditional computation (only the active unit and router incur inference cost). These mechanisms mirror those that enable the A320 to operate safely with degraded systems, the power grid to isolate faults without cascading failures, and the surgical robot to maintain control authority during network handover.

Chapter 4 identified interpretability deficits and latency unpredictability as the most blocking deployment gaps, motivating policy graphs' hard-routing call traces and commitment bounds respectively.

This chapter has three core contributions:

1. Policy graph formalism and training template (Contribution 1): We formalise policy graphs as directed graphs of callable policy units with explicit execution semantics (call-and-return, commitment bounds, constrained edges), and present a generic training template that supports modular data collection and updates. Policy graphs serve both as a learning structure—enabling skill specialisation and providing explainable routing decisions—and as a deployment framework that allows units to be distributed across different physical locations and hardware types, enabling System 1 impulses to execute on low-power edge devices near actuators whilst System 2 reasoning runs on remote GPU clusters.
2. Real-world proxy environments (Contribution 2): This chapter introduces BrowserEnv and FilesEnv, evaluation settings that deliberately couple simple, controllable dynamics with interface complexity characteristic of real-world deployment. BrowserEnv is used directly in the hard-routing study reported here, whilst FilesEnv broadens the interface setting and provides an additional proxy environment for future evaluation.
3. Two empirical construction routes (Contribution 3): First, this chapter shows that a competent monolithic teacher can be converted into a compact policy graph by clustering action-conditioned saliency traces into candidate behavioural regimes and distilling regime-specific specialists plus a router. Second, it evaluates hard attention routing over a fixed pool of specialists, with soft mixture-of-experts routing as a comparator, showing how the same policy-graph execution semantics can be realised when the unit inventory is fixed in advance.

The policy graph formulation synthesises insights from hierarchical reinforcement learning, real-world system design, and human skill acquisition. Chapter 2 established that division of labour—the principle that enabled Adam Smith's pin workers to achieve 240-fold productivity improvements—applies equally to learned control: specialised units coordinated through explicit mechanisms outperform monolithic approaches. Chapter 3 demonstrated how real-world systems (the A320's flight computers, the power grid's hierarchical control, the surgical robot's dual-console handover) embody these principles through redundancy, constrained transitions, and accountable delegation. Chapter 4 identified the deployment challenges (interpretability, latency predictability, safety constraints) that existing RL systems struggle to address. This section reviews the technical foundations that policy graphs build upon, connecting established hierarchical RL methods to the architectural patterns observed in engineered systems and the deployment demands identified in real-world applications.

Hierarchical reinforcement learning decomposes complex tasks into temporally extended subproblems, allowing policies to operate at multiple levels of abstraction—a computational analogue of the division of labour in Smith's pin factory (Chapter 2). The options framework formalises callable subpolicies with initiation sets and termination conditions [22], whilst feudal reinforcement learning emphasises hierarchical goal-setting between manager and worker levels [24]. These methods often require careful design choices around termination, subgoal representations, and skill priors, and can struggle to provide an implementation-level interface that supports flexible composition, constrained transitions, and deployment-oriented modularity.

Modular architectures provide an alternative route to specialisation: rather than imposing a fixed hierarchy, they learn to route computation through a subset of available modules. Mixture-of-experts models exemplify this idea by using a gating mechanism to select which expert(s) process a given input, trading dense computation for conditional activation [113]. In RL, similar routing mechanisms can be used to select among specialised encoders or policies on a per-state basis. A key practical distinction is between soft routing, which combines multiple modules in a weighted mixture, and hard routing, which selects a single module at a time. Hard routing enables three deployment-critical properties: first, simplicity—exactly one unit is responsible at any moment, making behaviour predictable; second, single-state efficiency—a routing decision can dispatch a single state to specific hardware without waiting for all units to complete processing; third, physical distribution—units can be separated across different locations and hardware types (low-power edge devices for reactive control, remote GPUs for compute-intensive reasoning) with routing determining which device becomes active. These properties align naturally with deployment constraints (latency, memory, and interpretability), but require explicit mechanisms to avoid collapse and unstable switching, which are central to the policy graph formulation.

A complementary line of work uses a strong teacher policy to guide the training of smaller or more structured students. Distillation transfers behaviour from teacher to student via imitation objectives (e.g., KL divergence between action distributions), optionally followed by RL fine-tuning [114]. In interactive settings, teacher-guided approaches are often paired with dataset aggregation methods, such as DAgger [115], that address covariate shift between expert and learner rollouts. For policy graphs, the central opportunity is to decompose the teacher's behaviour into reusable units with explicit interfaces and routing structure.

The motivation for modularity is not solely computational. Chapter 2 traced reward signals from ancient philosophy through modern neuroscience, culminating in Schultz's discovery that phasic dopamine spikes encode reward prediction error—the brain's mechanism for reinforcing successful actions and chunking them into reusable behavioural routines. Human skill learning exhibits a gradual progression from stimulus-driven responses to autonomous execution of behavioural chunks. Fitts and Posner's theory describes a transition from a cognitive stage (fragmented individual steps), through associative refinement (formation of chunks aided by dopamine reinforcement), to autonomous execution (refined chunks performed automatically with minimal conscious effort) [116][117]. These perspectives motivate a training strategy in which low-level policy units acquire reliable primitives from simple feedback—analogous to dopamine-driven chunking in the associative stage—whilst higher-level components learn to compose these primitives into long-horizon behaviour, mirroring the transition to autonomous skilled performance.

Taken together, existing HRL, modular routing, and teacher-guided learning provide powerful building blocks. However, they do not yield a single formulation that is simultaneously expressive (graph topologies), operational (explicit execution semantics), and implementation-ready (interfaces, buffers, and deployment constraints). Policy graphs are intended to fill this gap.

The policy graph formulation subsumes existing hierarchical RL approaches whilst addressing their practical deployment limitations. Options [22], feudal hierarchies [24], HAMs [26], and MAXQ [25] are each subsumed as special cases: options map to policy units with edge-encoded initiation sets; feudal managers map to routers; tree structures are relaxed to graphs that allow shared subskills and multiple callers. Soft MoE [113] is evaluated as a comparator in Section 5.7. Policy graphs unify these approaches whilst adding:

• Explicit delegation semantics (call-and-return) that make execution reproducible and debuggable.
• Graph topologies that generalise trees, enabling redundancy, shared subskills, and constrained transitions.
• Commitment and termination bounds that prevent unstable switching and provide worst-case guarantees, essential for real-world deployment.
• Modular training interfaces (unit-local buffers, call-level transitions) that support independent testing and swapping of components.
• Hard routing semantics that enable accountability, conditional computation, and physical distribution across heterogeneous hardware.

These properties are distilled from the architectural patterns of engineered systems examined in Chapter 3, providing a pathway from the operational clarity of those systems to the adaptability of learned policies.

A policy graph is a directed graph \(G=(V,E)\) whose nodes \(v\in V\) are callable policy units. Each unit implements a policy \(\pi_v\) (and optionally a value function or critic) that maps its inputs to either an environment action or a routing decision. Units may be trained with standard RL algorithms, including value-based methods such as DQN [12] and policy-gradient methods [14].

Edges \((u\rightarrow v)\in E\) represent permitted delegations: from unit \(u\), control may be transferred to unit \(v\) only if the corresponding edge is present. The routing decision can be implemented as an explicit router policy \(\pi_H\) (which selects the next unit), or embedded in the action space of the currently active unit; the formulation supports both, but the chapter emphasises hard-routing execution in which a single unit is active at any step.

Policy graphs require explicit interfaces to support modularity. At minimum, all units observe the current environment observation (or a shared embedding). In addition, transitions may carry structured information such as the caller identity, a compact memory state, or an “effect achieved'' flag that indicates whether a delegated objective was satisfied. These interfaces are intentionally lightweight: they are meant to be implementable and debuggable, rather than maximally expressive.

Policy graphs do not require an explicit notion of subgoal. In many environments, however, it is convenient to label delegations using goal-like or effect-like primitives: higher-level components can delegate what should be achieved rather than which low-level action should be taken next. This section briefly formalises goals and effects as optional interface choices used in parts of this chapter.

Goal-conditioned environments

RL environments are commonly formalised as Markov Decision Processes (MDPs). An MDP is a structure \(\mathrm{MDP}(\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R})\) where:

• \(\mathcal{S}\) is a set of states.
• \(\mathcal{A}\) is a set of actions.
• \(\mathcal{T}(s, s'| a) = \mathbb{P}(s_{t+1}=s'|a_t=a, s_t=s)\): The probability of a transition occurring between states \(s\) and \(s'\) when the agent takes action \(a\).
• \(\mathcal{R}(s, a, s') = \mathbb{E}(r_t|s_t=s, a_t=a, s_{t+1}=s')\): The expected reward gained when the system transitions from state \(s\) to \(s'\).

Goal-conditioned formulations augment the MDP with a goal variable. A \(\mathrm{GMDP}\) extends an MDP with a goal space \(\mathcal{G}\) and a goal-satisfaction relation \(\textsc{Sat} \subseteq \mathcal{S} \times \mathcal{G}\). Episodes are conditioned on a sampled goal \(g\in\mathcal{G}\), and rewards may depend on whether the current state satisfies the selected goal.

Goal-conditioned reward functions section:goalconditioned

In a goal-conditioned setting, a common choice is a sparse reward that agrees with the satisfaction relation, for example \(\mathcal{R}((s,a,s'),g)=1\) if \(\textsc{Sat}(s',g)\) and \(0\) otherwise (or the corresponding change-based variant when success is defined by reaching a newly satisfied state). Techniques such as HER exploit this structure by relabelling goals post hoc to extract learning signal from trajectories that do not achieve the originally sampled goal [20]. In policy graphs, goal labels can be used as part of the routing interface, but they are not required by the formulation.

Effects

Goals specify desired end-states, whereas actions specify immediate state transitions. For modular control, it is sometimes useful to define an intermediate primitive that represents a desired change relative to the state at which it is chosen. We call such a primitive an effect. Formally, an effect \(e\) can be represented as a relation on states, and is satisfied when the environment transitions from an origin state \(s_0\) to a state that stands in relation \(e\) to \(s_0\). Figure 5.4 illustrates several design choices for effect structure.

One way to train effect-conditioned behaviour is to use an origin-relative reward that fires when the effect becomes satisfied, as shown in Figure 5.5. In practice, the origin \(s_0\) is carried as part of the unit interface when an effect is selected, allowing the unit to detect effect satisfaction relative to the origin.

In policy graphs, effects can be used as routing labels: selecting an effect is treated as selecting a particular unit (or class of units) expected to realise that effect, and satisfaction information can be returned to the caller via an interface flag. This is an optional modelling choice. The remainder of the chapter adopts the more general execution semantics in which units delegate to other units directly, with effects and goals available when they provide useful structure.

Policy graph execution is defined by an explicit control-flow mechanism. At any time, there is a single active unit \(u_t\). Let \(\mathcal{A}_{env}\) denote the environment action space and let \(\mathcal{A}_{route}\) denote routing decisions (delegate/return). At each step, the active unit produces one of:

• an environment action \(a_t\in\mathcal{A}_{env}\), which is applied to the environment;
• a delegate-to decision selecting a successor \(v\) such that \((u_t\rightarrow v)\in E\); or
• a return decision, which transfers control back to the calling unit.

Delegation induces a call stack: when unit \(u\) delegates to unit \(v\), \(u\) becomes the caller of \(v\) until \(v\) returns. This call-and-return semantics makes the execution model reproducible and debuggable, and it aligns with common HRL patterns whilst allowing non-tree topologies through shared descendants and constrained transitions.

Commitment and termination

To avoid degenerate rapid switching, execution includes an explicit notion of commitment. In the default semantics used throughout this chapter, each invocation of a unit has a minimum and maximum duration \((k_{\min},k_{\max})\). The unit cannot return before \(k_{\min}\) steps, and must return (or be force-returned) after \(k_{\max}\) steps if it has not already delegated or returned. Learned termination functions \(\beta_v(s)\) can be used in place of (or in addition to) fixed bounds, but in all cases the execution engine enforces hard limits to ensure bounded rollouts.

Loop prevention

Since \(G\) may contain cycles, practical safeguards are required. We assume: (i) a maximum call-stack depth, (ii) per-invocation timeouts \((k_{\max})\), and (iii) optional switching penalties or hysteresis in the router objective. These mechanisms do not provide theoretical guarantees of loop freedom, but they yield predictable behaviour under realistic deployment constraints.

Policy graphs are intended to be trained with standard RL algorithms whilst retaining modularity. The key design choice is to make data and updates unit-local wherever possible.

Data collection

When unit \(v\) is active, its interactions with the environment are recorded in a unit-specific buffer (e.g., a replay buffer for off-policy learning). In addition, the router (or caller) can record boundary transitions at delegation and return events, including the identity of the callee, the cumulative reward accrued during the callee's execution, and termination information (timeout vs explicit return). This produces two complementary datasets: fine-grained environment transitions for training units, and coarse-grained call-level transitions for training routing policies.

Update schedule

Joint training induces non-stationarity because units and router co-adapt. A practical template is to alternate between (i) updating units using their local buffers under the current routing distribution, and (ii) updating the router using call-level transitions under (partially) stabilised units. Freezing subsets of units for short windows can further reduce drift when the router is learning rapidly.

Encouraging specialisation

Modularity is only useful when different units adopt distinct roles. One pragmatic approach is to initialise a pool of units using diverse auxiliary rewards. We refer to these as divergent rewards: rewards that admit multiple high-return behaviours and thereby encourage a heterogeneous set of skills, in contrast to goal-specific rewards that tightly constrain behaviour. Divergent rewards are related to intrinsic-motivation objectives [56][118]. In the policy-graph context, such rewards are best viewed as an initialisation mechanism; subsequent training aligns units with the tasks they are actually assigned by the router.

The graph structure is not merely a notational convenience. Relative to trees or flat sets of options, graphs support shared subskills (a unit may have multiple callers), constrained transitions (edges encode permissible handoffs), and non-tree topologies that better reflect real execution constraints. These properties are useful both for learning and for deployment: units can be trained, tested, swapped, and cached independently, and routing constraints provide a natural place to encode safety rules or interface limitations. Critically, graphs enable distributed execution: policy graphs function both as a learning structure (skill specialisation, explainable routing decisions) and as a deployment framework (units distributed across networks, different physical locations, hardware-specific devices). For instance, System 1 impulses—rapid, reactive responses—can execute on low-power edge hardware near actuators, minimising latency for time-critical actions. Meanwhile, System 2 reasoning—deliberate, compute-intensive planning—runs on remote GPU clusters with abundant computational resources. This separation mirrors the architectural principles observed in real-world systems (Chapter 3): the power grid's IEDs handle immediate local faults whilst SCADA coordinates higher-level nationwide decisions. The systems-level scheduling and placement questions are fully addressed in Chapter 8, which extends the framework to network-aware training and deployment.

Table 5.1 maps each design principle observed in Chapter 3 to its policy graph realisation and the real-world analogue that motivates it.

By embedding these principles as first-class components, policy graphs provide a pathway for reinforcement learning to inherit the operational clarity of engineered systems whilst retaining the adaptability of learned policies.

section:eval_envs section:browserenv

Many of the core design choices in this chapter—hard routing, explicit commitment, unit-local buffers, and call-and-return traces—are motivated by the practicalities of deploying agents in interactive computing environments. A substantial class of deployment-relevant problems is characterised by the need to act through high-dimensional interfaces with long horizons and discrete, stateful structure. Web browsing is a useful proxy for this regime: the transition dynamics induced by mouse and keyboard events are straightforward to implement and instrument, yet successful behaviour requires robust perception, precise low-level control, and the composition of many small interactions into coherent workflows.

Browser environments exhibit long horizons, high-dimensional observations, and sparse rewards—characteristics that motivate the policy graph framework: modular units can specialise in distinct interaction regimes (navigation, form-filling, content extraction), whilst explicit routing and commitment bounds provide the structure required for interpretable, debuggable behaviour.

BrowserEnv is a Gymnasium-compatible environment that exposes a real browser instance to an RL agent. Each environment instance runs Firefox inside a Docker container configured with a fixed display resolution and a controlled profile. The containerised design supports parallel training by allocating each instance a static IP on an isolated Docker network, as illustrated in Figure 5.7. Agents interact with the browser through low-level input primitives. In the reference implementation, these inputs are realised via a VNC connection: a client issues mouse movements and clicks (and, when required, keyboard events) and captures screenshots of the rendered viewport. This design keeps the environment mechanics simple, whilst maintaining the interaction bottlenecks that matter in practice: pixel-level perception, delayed feedback, and long-horizon credit assignment.

A lightweight browser extension provides structured instrumentation in addition to pixels. The extension forwards navigation events and records interaction signals such as the text and bounding rectangle of clicked elements, scroll deltas, link-hover notifications, and text selections extracted as complete sentences. It also enumerates hyperlinks on page load, which enables bookkeeping of previously observed URLs and supports curricula that reset to pages discovered earlier in training. These signals are surfaced to the agent through the environment's info dictionary alongside the current URL and simple flags indicating whether a navigation occurred and whether the page was novel within the current episode.

The observation and action interfaces are designed to support both end-to-end and modular approaches. Observations may be taken as full RGB frames of the browser viewport, or as a foveated crop centred on the current cursor location, padded where necessary. The latter provides a compact observation that reduces input dimensionality whilst requiring active scanning. Actions may similarly be specified either as a discrete set of relative cursor nudges with a click action, or as absolute \((x,y)\) coordinates for pixel-precise pointing. In both cases, the intent is to provide an interaction substrate that is compatible with standard RL libraries whilst remaining faithful to the practical constraints of GUI control.

The default BrowserEnv reward is intentionally simple. It provides an exploration-style shaping signal by rewarding discovery of previously unseen pages and domains, and includes small incentives for interactions that reflect content engagement, such as meaningful scrolling or non-trivial text selection. This shaping is not intended to define a single canonical task; rather, it provides a lightweight scaffold for learning stable interaction primitives in an environment where sparse objectives are otherwise difficult to specify. In downstream settings, the same environment can be paired with task-specific reward and termination criteria, either by modifying the environment or by wrapping it to consume the rich event stream exposed by the extension.

Practical safeguards are included to maintain robustness during long runs. For example, the environment can detect stale sessions in which no messages are received for an extended period and can trigger a clean reconnection. The implementation of BrowserEnv is released in an open-source capacity.https://github.com/StandardRL-Components/BrowserEnv

As a smaller companion environment, we also provide FilesEnv, which applies the same containerised, VNC-driven approach to interaction with a desktop file manager. Typical tasks involve browsing directories, selecting files, and carrying out simple multi-step file operations. FilesEnv therefore broadens the scenario set beyond web navigation without carrying the same empirical weight in this chapter. Taken together, the two environments offer a practical substrate for studying modular policies for general computer interaction, whilst preserving the controllability and instrumentation required for systematic evaluation.

The policy graph formulation is intended to be a practical interface, not merely a descriptive framework. A central question is therefore how to obtain a useful graph: how to define units, how to define routing, and how to train them jointly under realistic constraints. This chapter presents two complementary construction recipes. Both are motivated by the same observation surfaced by BrowserEnv and FilesEnv: in interface-rich environments, routing decisions and their traces are an operational requirement for debugging, reliability, and compute control.

The first recipe is teacher-guided graph synthesis in which a strong teacher policy provides trajectories and attribution signals that are used to discover behavioural regimes; these regimes define candidate units, which are then trained and routed in a compact student graph. The second recipe assumes a fixed pool of specialist modules and focuses on learning a robust hard-routing mechanism with explicit commitment and regularisation, whilst comparing against soft mixture-based routing under matched budgets. Both recipes instantiate the same template: nodes (units), routing (a router or embedded decisions), commitment/termination, and a training objective that balances task performance against stability and efficiency constraints.

The first construction route answers a question left open by the fixed-specialist setting: where should units themselves come from? In many deployment-relevant domains the difficult part is not routing between known specialists, but discovering a specialist inventory without hand-labelling subtasks. This section develops a teacher-guided answer: a competent monolithic teacher generates trajectories and action-conditioned saliency traces; recurring saliency structure is treated as evidence of candidate behavioural regimes, and these regimes are distilled into the units of a compact student graph. The route complements the hard-routing study in Section 5.7—the present section addresses unit discovery, the later section addresses routing robustness—and both share the same policy-graph semantics from Section 5.3.

The intended deployment setting is interface-rich control (BrowserEnv, FilesEnv); the empirical treatment here uses MiniGrid as a controlled proof of concept [119], designed for later extension to BrowserEnv and FilesEnv.

Let \(\pi_T\) denote a frozen teacher policy and let \(\mathcal{D}=\{(o_t,a_t,r_t)\}\) denote trajectories generated by rolling out \(\pi_T\). At each step we compute an action-conditioned saliency map \[ S_t = Norm\!(| _T(a_t o_t) o_t|), \] where \(a_t\) is the action chosen by the teacher and \(\mathrm{Norm}\) denotes per-frame normalisation. The working hypothesis is deliberately modest: these maps need only function as a practical signal of what parts of the observation matter when the teacher behaves in different ways. They are used as a regime-discovery feature, not as a proof of deep causal interpretability [120][121].

Teacher, saliency, and regime discovery. For each teacher rollout step, the saliency map retains the same channel and spatial structure as the observation. The maps are flattened, projected with PCA to retain \(95\%\) of variance, and clustered with \(K\)-means over \(K\in\{2,3,4,5,6\}\). Because raw cluster assignments flicker near behavioural boundaries, we smooth them independently within each episode using a majority filter (window \(W=5\)) followed by a minimum-segment-length merge (\(L_{\min}=3\)). The resulting labels \(\bar{z}_t\) are treated as candidate behavioural regimes: recurrent attribution patterns coherent enough to support specialist construction, but not claimed to be the task's true latent options.

From regimes to policy-graph units. Each regime \(k\) is mapped to a specialist unit \(\pi_k\) with training dataset \[ D_k=\(o_t,a_t)D z_t = k\. \] The router supervision signal is the smoothed label sequence \(\{\bar{z}_t\}\). The resulting student graph is a flat policy graph with one router and \(K\) specialists. At invocation boundaries the router selects a specialist; the selected specialist then executes for a fixed commitment horizon \(H\) before routing may be reconsidered. In other words, the discovered regimes are not merely descriptive clusters: they become the nodes of an executable policy graph under the same call-and-return and commitment semantics defined in Section 5.3.

Training schedule and saliency validation. Each specialist is pretrained by KL distillation on its regime-specific dataset, \[ L_spec^(k) = E_o D_k\!, \] with inverse-frequency weighting so that rare regimes are not ignored. The router is pretrained by cross-entropy on the smoothed regime labels. The full graph is then fine-tuned with PPO [122], using an auxiliary imitation term and a small router load-balancing penalty adapted from mixture-of-experts training [123]. Before relying on saliency for clustering, we perform a simple masking validation: the top \(20\%\) of salient input components are masked on held-out evaluation rollouts and compared against random masking at the same fraction. The intention is only to confirm that saliency carries decision-relevant structure; stronger interpretability claims are unnecessary for the present construction route.

The primary benchmark is MiniGrid-KeyCorridorS3R3 [119]. The agent observes a \(7\times7\times3\) egocentric grid and must locate a key, collect it, navigate to the correct locked room, open the door, and reach the target object. This makes KeyCorridor a useful main environment for the synthesis route: behaviour is clearly multi-stage, yet the observation space remains small enough for saliency extraction, clustering, and visualisation to be reproducible. Three auxiliary environments are also used. FourRooms provides a simpler two-phase navigation problem; UnlockPickup provides a longer dependency chain; and MemoryS13 is used more narrowly as a saliency diagnostic, asking whether the pipeline identifies the memory-critical token in a task where the teacher itself remains near chance.

For KeyCorridor, the teacher is a compact convolutional PPO policy with two convolutional layers (16 and 32 filters, \(2\times2\) kernels), a 64-unit fully connected layer, and a seven-action output head. The teacher is trained for \(3\) million environment steps across three seeds; the best checkpoint under a short validation run is frozen, then re-evaluated over 200 episodes to obtain the authoritative teacher reference of \(21.5\%\) success. Specialists reuse the same backbone but are independently parameterised. The router is a lightweight two-layer MLP operating on a shared convolutional encoder. The default fine-tuning horizon is \(H=10\).

The baseline set is intentionally compact. The most important comparator is a monolithic student matched in total parameter count to the full specialist pool and distilled from the teacher on the full dataset. Additional baselines test whether any segmentation would suffice: random regime assignments, clustering on raw observations, clustering on teacher hidden states, and a saliency pipeline without temporal smoothing. On KeyCorridor and UnlockPickup we also compare against a DDO-style latent-variable segmentation baseline based on an HMM fitted to PCA-embedded observation sequences [124][125]. The key question is not whether the graph must beat the teacher, but whether it remains viable as an explicit modular controller where compact monolithic distillation does not.

Teacher saliency exposes coherent candidate regimes

Figures 5.9--5.11 show the three pieces of evidence needed for regime discovery. First, the saliency maps themselves vary qualitatively across phases of behaviour. Second, the PCA projection suggests that these attribution patterns occupy partially distinct regions in embedding space even after aggressive dimensionality reduction. Third, the labels persist across multi-step segments within an episode rather than oscillating chaotically at every timestep.

The masking validation supports the use of saliency as a clustering feature, though only in the bounded sense required here. On FourRooms, masking the top-saliency region collapses success from the mid-fifties to \(4\%\), whereas random masking at the same fraction leaves success around \(39\%\). On MemoryS13 the contrast is sharper still: top-saliency masking reduces success to \(0\%\), whilst random masking leaves it around \(53\%\). On KeyCorridor and UnlockPickup both masking strategies are highly destructive, which is itself informative: in these compact control tasks the teacher depends on much of the frame at once, so masking is too blunt an instrument to serve as a causal test. Even there, however, the relative saliency structure across timesteps remains rich enough to support regime discovery.

Table 5.3 highlights a structural tension that recurs throughout this section: the silhouette-selected construction is usable, but the discovered regimes are uneven in size. In particular, the key-pick-up regime occupies only about \(3\%\) of all frames. This does not invalidate the construction route, but it helps explain why cluster quality and downstream control quality are not identical objectives.

KeyCorridorS3R3 yields a viable graph where monolithic distillation does not

The main no-label result is the silhouette-selected \(K=5\) graph in Table 5.4. On that construction, the routed student reaches \(28.5\%\pm7.4\%\) success, modestly above the teacher reference of \(21.5\%\), whilst the parameter-matched monolithic student fails completely under both standard and teacher-level hyperparameters. This is the central empirical point of the section: structured specialist decomposition remains workable in a setting where naïve monolithic distillation does not. The weaker decomposition baselines also fall clearly below the saliency graph, and removing temporal smoothing is particularly harmful, reducing success to \(7.0\%\pm3.4\%\).

Figure 5.12 shows that the discovered graph is not merely numerically viable but structurally inspectable. The router activates different specialists during room transitions, key search, key collection, corridor navigation, and goal approach, producing a trace that can in principle be logged, debugged, or routed onto different hardware in later systems chapters.

Figure 5.13 and Table 5.5 clarify an important nuance. The silhouette criterion selects \(K=5\), which yields a viable graph and is therefore the honest no-label construction result. Downstream control, however, peaks at \(K=2\) with \(54.2\%\pm3.4\%\) success. In practice this means the clustering score is a sensible starting point rather than an oracle: cluster separation and control utility are related, but not identical. The commitment ablation tells a more stable story. With no commitment (\(H=1\)), success drops to \(24.0\%\) and collapse worsens; intermediate horizons around \(H=5\)--\(10\) are clearly better, which supports the chapter-wide argument that bounded commitment is not merely a formal embellishment but a practically stabilising design choice.

The HMM comparison is also revealing. On KeyCorridor, a DDO-style HMM segmentation reaches \(45.7\%\pm4.3\%\) at \(K=2\): much stronger than the clustering baselines, but still below the saliency graph at the same \(K\). The implication is not that saliency is universally superior, but that teacher attribution can add useful signal when behavioural modes differ more in what the teacher attends to than in the raw appearance of the frame.

Auxiliary environments, limitations, and thesis linkage

The auxiliary environments help delimit the claim. On FourRooms, the saliency graph slightly exceeds the teacher, but raw-observation and hidden-state clustering are already close, which is consistent with a simpler two-phase task whose structure is visible directly in appearance space. On UnlockPickup, the saliency graph reaches \(94.7\%\pm1.0\%\) success, yet an HMM baseline performs almost identically. Here the large gain appears to come primarily from the downstream specialist-and-router pipeline rather than from saliency alone. MemoryS13 should be read differently again: the teacher is a memoryless MLP operating near chance, so the point is not that the graph outperforms it, but that saliency correctly identifies the memory-critical observation token.

These results expose the main limitations of the route. The method depends on teacher quality: if the teacher is inconsistent, the regime labels inherit that inconsistency. Gradient saliency is sensitive to representation choice and does not by itself prove causal necessity [126]. Diffuse saliency on compact observations can make masking uninformative even when relative saliency patterns still support clustering. Regime boundaries remain brittle, and the route is still demonstrated here only in controlled MiniGrid tasks rather than directly in BrowserEnv or FilesEnv. Those limitations should narrow the claim, not erase it. What this section establishes is that policy graphs need not assume a hand-specified unit inventory: teacher behaviour can itself be used to synthesise candidate units and a router under the operational semantics of this chapter.

This fills the first of the two construction routes promised at the start of Chapter 5. The section below turns to the complementary problem. If a specialist pool is already available—whether hand-designed, inherited from prior work, or discovered by the synthesis route above—how should routing be learned, regularised, and compared against soft mixtures under matched budgets?

This section presents the second construction recipe outlined in Section 5.5: hard attention routing over a fixed pool of specialist policies. This chapter instantiates the policy-graph execution semantics defined in Section 5.3—single active unit, explicit commitment, call-and-return traces—and evaluates whether hard routing improves stability, interpretability, and conditional-compute efficiency relative to soft mixture-of-experts (MoE) baselines across ViZDoom, Procgen, and BrowserEnv. Hard routing is compared against soft MoE under matched parameter budgets and a compute-matched top-\(k\) soft baseline to isolate the effect of softness from compute; all experiments report compute proxies alongside performance and interpretability metrics.

In long-horizon visual control, a single end-to-end policy must simultaneously learn perception, control, and regime-dependent behaviour selection. This often yields high variance across seeds, brittle boundary behaviour, and inference costs that scale with the full model even when only a subset of computation is relevant at a given moment. Policy graphs provide an implementation-ready abstraction (Section 5.3) in which reusable policy units are composed with explicit call-and-return semantics and bounded commitment, making routing decisions inspectable and deployment constraints enforceable—properties particularly valuable in the interface-rich settings exemplified by BrowserEnv (Section section:browserenv).

This section focuses on the construction recipe described in Section 5.5: hard attention routing over a fixed pool of specialists, with soft routing (mixtures) treated as a strong comparator. Our central question is: quote Given a fixed pool of specialist policy units, can we learn a router that selects one unit at a time with explicit commitment, and how does this compare to soft mixtures under matched budgets? quote

The investigation connects to broader thesis themes: the division-of-labour principles established in earlier chapters motivate specialisation, whilst the efficient edge models developed in Chapter 7 and the distributed infrastructure provided by Chapter 8 enable deployment of such modular systems across heterogeneous hardware.

Policy graph instantiation, hard routing, and commitment

This chapter instantiates a two-level policy graph: a router (manager) delegates to one of \(K\) specialist policy units, each of which executes for multiple environment steps before returning control. Only a single specialist is active at any time. At a call boundary, the router outputs logits \(z=g_{\theta}(s)\in\mathbb{R}^{K}\) and samples specialist index \(i\sim\mathrm{Cat}(z)\); the selected unit executes environment actions \(a\sim\pi_{\phi_i}(a\mid s)\) until it returns. Each invocation obeys the explicit commitment bounds \((k_{\min},k_{\max})\) from Section 5.3; in the primary experiments we use fixed-horizon calls \(k_{\min}=k_{\max}=H=10\) (ablated in Section 5.7). The router is trained with PPO on macro-transitions \((s_{\text{call}}, i, r_{\text{call}}, d, s_{\text{return}}, \Delta t)\) using discount \(\gamma^{\Delta t}\); each specialist is trained with PPO on its unit-local step buffer.

Training objectives and anti-collapse

Hard routing risks collapse: one unit dominates whilst others fail to specialise. This chapter uses a usage-threshold penalty on the router's action distribution (minimum usage 0.10, maximum usage 0.40; underuse weight 5.0, overuse weight 10.0, coefficient 5.0) plus an optional switching penalty at delegation boundaries (ablated). The soft MoE comparator replaces discrete delegation with per-step mixture weights \(w(s)=\mathrm{softmax}(g_{\theta}(s))\), sampling from \(\pi_{\text{mix}}(a\mid s)=\sum_{i} w_i(s)\,\pi_{\phi_i}(a\mid s)\); a compute-matched soft-top-\(k\) variant (with \(k=2\)) isolates the effect of softness from compute cost.

Each policy unit (and the router) employs a CNN backbone (conv(32, \(8{\times}8\), s4) \(\to\) conv(64, \(4{\times}4\), s2) \(\to\) conv(64, \(3{\times}3\), s1) \(\to\) linear(256)) matched to the efficient architectures discussed in Chapter 7, with MLP heads for policy logits, value, and routing. Table 5.7 summarises the per-environment preprocessing.

Training uses PPO with the following hyperparameters:

• Optimiser: Adam, learning rate \(3{\times}10^{-4}\), \(\epsilon=10^{-5}\), gradient clipping 0.5.
• Discounting: \(\gamma=0.99\), GAE \(\lambda=0.95\).
• PPO: clip range 0.2, value coefficient 0.5, entropy coefficient 0.01, PPO epochs 4.
• Minibatch size: 32 (primary), with optional replication at minibatch size 64.
• Rollout/update interval: 2048 environment steps per update.
• Evaluation: every 30,720 environment steps, 3 episodes, maximum length 2000, greedy (deterministic) action selection.
• Training horizon: 1,000,000 environment steps per scenario per seed (primary), with optional 10,000,000-step extended runs.

After each rollout, we update (i) each specialist on its unit-local step buffer and (ii) the router on the call-level buffer, using the same PPO hyperparameters. BrowserEnv uses the same hyperparameters but a reduced budget of 200,000--500,000 steps per seed to reflect its higher wall-clock variability; this budget is reported explicitly alongside results.

Benchmark set

Results are reported on three deliberately diverse domains:

• ViZDoom (3D partial observability): scenarios basic, deadly_corridor, health_gathering, defend_the_centre.
• Procgen (procedural 2D): games heist and coinrun with the default difficulty distribution; frame stacking introduces partial observability.
• BrowserEnv (realistic UI interaction): the environment introduced in Section section:browserenv, run in zoomed observation mode to maintain comparable input sizes. This setting probes transfer-relevant failure modes and instrumentation needs.

For each environment configuration, we use \(K=6\) specialists and report results across 3 random seeds.

Budget reporting

Reported metrics include (i) parameter counts and (ii) compute proxies as expert forward passes per environment step: hard routing uses \(\approx 1\) expert forward per step plus router passes every \(H\) steps; soft MoE uses \(K\) expert forwards per step (or \(k\) for soft-top-\(k\)), enabling hardware-independent comparison.

Evaluation covers the dimensions identified in the Conclusion (Section 5.8):

• Performance: average return versus environment steps (per scenario).
• Stability: variance/dispersion across seeds (standard deviation and interquartile range) for learning curves and final evaluation.
• Efficiency (hardware-independent): expert forward passes per environment step; router forward passes per step.
• Efficiency (optional): wall-clock frames per second and latency, reported only alongside the exact hardware and software stack used.
• Interpretability: specialist usage entropy; switch rate; call duration distribution; forced returns due to commitment violations.

These metrics directly instantiate the empirically testable benefits discussed in Section 5.8, providing grounding for the efficiency, stability, and interpretability motivations.

Systematic ablations cover the key components of the policy-graph formulation:

• Commitment horizon \(H\in\{5,10,20\}\): characterises the commitment-stability trade-off.
• Anti-collapse coefficient \(\lambda_{\text{ac}}\in\{0,5\}\): tests the necessity of usage-threshold penalties.
• Number of specialists \(K\in\{3,6,9\}\): explores the specialisation-coordination trade-off.
• Soft compute matching: full MoE versus top-\(k\) with \(k=2\).
• Switching penalty: on/off comparison at boundaries.

Hard routing over specialists achieves comparable task performance to soft MoE baselines whilst providing improvements in computational efficiency, cross-seed stability, and interpretability. All experiments use \(K=6\) specialists with commitment horizon \(H=10\) unless otherwise specified, trained for 1M environment steps across 3 random seeds; stability claims should be read as bounded by this three-seed protocol.

Main Performance Comparison

Table 5.8 presents performance across ViZDoom scenarios and Procgen games. Hard routing achieves 94.3% of soft MoE performance on average whilst requiring only 16.7% of the expert forward passes (1.0 vs.\ 6.0 per step). The compute-matched soft-top-2 baseline (using 2.0 expert forwards per step) achieves intermediate performance at 96.8% of full soft MoE, validating that the performance gap is primarily attributable to reduced compute rather than the discreteness of routing decisions.

Within this three-seed study, hard routing exhibits substantially lower variance across seeds: the mean standard deviation across all environments is 7.2 for hard routing versus 13.0 for soft MoE and 10.6 for soft-top-2. This stability improvement is most pronounced in high-variance scenarios such as Deadly Corridor (std 9.3 vs.\ 18.7) and Defend the Centre (std 11.6 vs.\ 21.4), where the commitment mechanism prevents rapid switching between specialists that can destabilise learning.

Computational Efficiency Analysis

Table 5.9 quantifies the computational savings achieved through hard routing. By activating only a single specialist per environment step (plus router overhead every \(H=10\) steps), hard routing reduces expert forward passes by 83.3% relative to full soft MoE whilst maintaining 94% of task performance.

The router forward pass frequency of 0.10 per step reflects the commitment horizon: routing decisions occur every 10 steps, amortising the delegation overhead. This enables deployment scenarios where specialists execute on heterogeneous hardware (edge processors for reactive control, cloud GPUs for planning) whilst minimising inter-device communication frequency—a critical requirement for the distributed policy graph execution explored in Chapter 8.

Interpretability and Routing Behaviour

Table 5.10 presents routing behaviour metrics. Hard routing achieves low usage entropy (0.87 \(\pm\) 0.14 across environments), indicating strong specialisation: specialists concentrate on distinct subsets of state space rather than blending uniformly. The switch rate of 0.094 per step closely matches the theoretical maximum of \(1/H = 0.10\), confirming that commitment bounds are actively enforced and specialists complete their assigned horizons without premature returns.

The percentage of forced returns (episodes where \(k_{\max}\) is reached and return is mandated) ranges from 2.8% to 9.1%, indicating that specialists typically complete their objectives within the commitment window and return control voluntarily. Higher forced return rates in Deadly Corridor (8.7%) and Defend the Centre (9.1%) reflect these scenarios' complex, multi-phase structure, where specialists occasionally require the full commitment duration to complete local objectives.

For comparison, soft MoE exhibits usage entropy of 1.21 \(\pm\) 0.09 (closer to uniform distribution over \(K=6\) specialists: \(\log(6) \approx 1.79\)), indicating less pronounced specialisation. The hard routing advantage in interpretability manifests as discrete call traces: at any moment exactly one specialist is responsible, producing human-readable delegation sequences such as “Specialist 2 (navigation) \(\rightarrow\) Specialist 5 (combat) \(\rightarrow\) Specialist 2 (navigation)''.

Ablations

Table 5.11 summarises the commitment-horizon and specialist-count sweeps. The default \(H=10\) balances switching stability and adaptability: shorter horizons increase variance, longer horizons reduce adaptability. Performance improves from \(K=3\) to \(K=6\) but shows diminishing returns at \(K=9\). Removing anti-collapse penalties (\(\lambda_{\text{ac}}=0\)) causes usage entropy to collapse to \(0.34 \pm 0.21\) and degrades performance by 23% on average, confirming that balanced utilisation requires explicit regularisation.

BrowserEnv Transfer Evaluation

On BrowserEnv form-filling tasks (200K training steps, limited budget), hard routing achieves 38.2% success rate versus 41.7% for soft MoE. Routing patterns reveal interpretable specialisation: Specialist 1 focuses on text input fields (62% activation on form states), Specialist 4 handles button interactions (71% activation on submit states), and Specialist 3 manages scrolling and navigation (58% activation on multi-page forms). Under this limited-budget protocol, these patterns provide suggestive rather than definitive evidence that policy graphs can discover task-relevant decompositions in complex interface environments.

However, BrowserEnv exhibits substantially higher variance (std 18.3 for hard routing vs.\ 12.7 for ViZDoom average), reflecting the environment's sensitivity to rare interaction sequences and the limited training budget. Failure mode analysis indicates that forced returns occasionally interrupt multi-step interaction sequences (e.g., filling form field \(\rightarrow\) submit button requires two specialists, but commitment forces return mid-sequence), suggesting that learned termination functions \(\beta_i(s)\) or task-conditioned commitment horizons could improve coordination in such settings.

Hard routing improves modular isolation, conditional computation, and accountability: only one unit is responsible for actions over a committed segment, making failures localisable and trajectories readable as call sequences. This directly implements the execution semantics and training template defined in Section 5.3. Soft mixtures provide smoother optimisation and can blend behaviour at ambiguous boundary states, but obscure which unit is responsible for an action and can be more expensive at inference if all experts are evaluated—a critical consideration for edge deployment (Chapter 7) and distributed execution (Chapter 8).

Transfer to real-world environments

In real environments such as BrowserEnv, regimes are heterogeneous and only weakly labelled; routing therefore becomes an implicit interface choice rather than an explicit goal-conditioned primitive (Section 5.3). Commitment and enforced timeouts become reliability mechanisms: they prevent unstable switching, bound worst-case behaviour, and provide deployment guarantees essential for real-world systems. Critically, instrumentation is part of the method: routing decisions, call durations, forced returns, and switch triggers must be logged to debug failures. Soft mixtures may reduce accountability, which complicates deployment debugging compared to hard call-and-return traces.

This observation connects to the lessons from Chapter 3, where real-world systems (aviation autopilots, medical devices) employ explicit handoffs and accountability mechanisms for safety-critical operation. Policy graphs extend these principles to learned systems.

Connection to distributed deployment

The hard-routing architecture naturally supports the distributed policy-graph deployment infrastructure developed in Chapter 8: each specialist can execute on a different device (edge processor, cloud server, GPU accelerator), with routing decisions determining which device is active. The commitment mechanism bounds communication overhead (at most one handoff every \(H\) steps), whilst the call-and-return traces provide the accountability required for debugging distributed failures. Chapter 8 extends Contribution 1's training template to network-aware learning, where latency, jitter, and packet loss become environmental properties that the router must learn to navigate—mirroring how the power grid's SCADA system (Chapter 3) coordinates IEDs across diverse network conditions. The systems-level implementation explores how heterogeneous hardware placement (edge units for low-latency perception, cloud units for compute-intensive planning) can be managed whilst preserving the operational guarantees established in this chapter. This points towards a more operational pathway: from the formalism presented here, through the network-aware training of Chapter 8, and onward to the initial hardware realisation sketched in Chapter 9.

This chapter uses fixed-horizon commitment (\(k_{\min}=k_{\max}=H\)) for clarity and stability; learning termination functions \(\beta_i(s)\) (as outlined in Section 5.3) is an important extension, with the policy-graph execution engine still enforcing hard bounds to maintain deployment guarantees. More expressive graph topologies (beyond a flat set of specialists) and constrained transitions could improve compositionality, enabling richer sharing patterns as suggested in Section 5.3. Finally, distilling a soft MoE into a hard router for deployment—potentially using the teacher-guided decomposition recipe developed in Section 5.6 as a front-end for unit discovery—is a natural next step that would further unify the two construction approaches presented in this chapter.

Policy graphs distil the architectural principles of real-world systems—specialisation, constrained transitions, commitment bounds, redundancy, accountability—into a deployment-oriented framework for modular reinforcement learning. The formulation targets an operational gap left by much existing HRL work: execution semantics that can be implemented, inspected, and constrained during deployment. Options, feudal hierarchies, and mixture-of-experts provide temporal abstraction, but lack call-and-return traces, commitment bounds, constrained edges, and modular interfaces. Policy graphs embed these as first-class components, inheriting patterns from the A320's flight computers, the French power grid's hierarchical control, and the Kangduo surgical robot's dual-console handover.

The chapter makes three core contributions:

1. Policy graph formalism and execution semantics: Hard routing over \(K=6\) specialists achieves 94.3% of soft MoE performance at 6\(\times\) lower compute and 1.8\(\times\) lower cross-seed variance, with call traces that provide explicit unit-level accountability. Saliency-guided synthesis discovers a viable student graph in KeyCorridorS3R3 where parameter-matched monolithic distillation fails completely, demonstrating that the formalism supports practical construction from teacher behaviour.
2. Dual role as learning structure and deployment framework: Unit-local buffers enable specialisation whilst graph topology encodes deployment constraints (co-location, bandwidth, network tolerance). System 1 impulses execute on low-power edge devices near actuators; System 2 reasoning runs on remote GPU clusters. Edges encode both logical dependencies and deployment constraints; commitment bounds control communication overhead; call traces enable reconstruction of distributed failures.
3. Two complementary construction routes: The first route shows that a competent monolithic teacher can be converted into a compact policy graph by clustering action-conditioned saliency traces into candidate behavioural regimes, distilling regime-specific specialists, and training a router under commitment-bounded execution. The second route studies the complementary fixed-specialist problem: hard attention routing over an existing pool of units, compared against soft mixtures in BrowserEnv, ViZDoom, and Procgen. Together, the two routes address both sides of policy-graph construction: discovering units and stabilising routing once those units exist.

Common failure modes—collapse, handoff errors, non-stationarity, loops—mirror real-world system failures. Policy graphs address these through usage-threshold penalties, commitment bounds with hysteresis, unit-local buffers with alternating updates, and call-stack depth limits with timeouts. This design philosophy—make failures explicit, provide bounded recovery, maintain interpretable traces—distinguishes policy graphs from approaches that treat modularity as optimisation rather than operational requirement.

Six empirically testable benefits align with real-world deployment requirements: efficiency via conditional computation, stability via commitment bounds, isolation via modular training, interpretability via call traces, deployment hooks via constrained edges and timeouts, and distributed execution readiness via commitment-bounded handoffs. Soft routing blends multiple units simultaneously, sacrificing accountability and conditional-compute benefits for potentially smoother credit assignment—Section 5.7 quantifies these trade-offs empirically under matched budgets.

Whilst single-machine experiments demonstrate learning properties—specialisation, stability, interpretability—Chapter 8 takes up the systems question of network-aware learning across heterogeneous hardware, incorporating latency and jitter into routing objectives and exploring simple distributed deployments. Together, Chapters 5 and 8 provide a pathway from formalism towards operational deployment. Open challenges remain: automatic discovery of richer effect interfaces, formal termination guarantees, broader validation of teacher-guided synthesis in interface-rich environments such as BrowserEnv, and tighter coupling between discovered graphs and hardware-aware execution. Despite these, policy graphs provide operational semantics that are implementable and debuggable—a principled pathway from learned adaptability to engineered accountability.