Principal-agent problems in automation in cloud infrastructure

The economist who named the gap

In 1963, Kenneth Arrow was trying to explain why medical insurance markets behaved so strangely, and landed on a structural fact: the patient cannot verify what the doctor did or knew. Akerlof, in 1970, showed what that gap does to a market when it goes unaddressed — sellers of poor-quality cars know their cars are poor, buyers cannot tell, and the good cars leave the market first. Ross gave the general problem its name in 1973. Jensen and Meckling, in 1976, turned it into a theory of the firm: shareholders as principals, managers as agents, governance as the cost of narrowing what managers can hide. Holmström's 1979 informativeness principle supplied the design rule that survives to this day — a contract should condition on any signal that carries information about what the agent actually did.

Underneath all four papers sits the same precondition, rarely stated because it was rarely in doubt: the agent has to be capable of reporting something the principal can check. A doctor can describe a diagnosis. A manager can produce accounts. Monitoring, incentive pay and disclosure rules all presuppose an agent with the internal representation needed to disclose. Take that precondition away and none of the remedies function. You are no longer contracting. You are trusting.

The dependency that used to be reliable

Cloud infrastructure runs on delegation almost by definition. An SRE is a principal who has handed operational judgement to systems that watch service telemetry, trace dependency graphs, ingest deploy events and correlate incident feeds, because no human can hold a live picture of several hundred interacting services in their head. The automation is the agent. The question Arrow's framework asks of any agent applies here without translation: does the agent know something the principal cannot see, and can it say so?

The characteristic failure in this domain has a specific shape. A downstream dependency — a caching layer, a third-party API, an internal service two hops away — was reliable for eighteen months. Its reliability profile changed: a maintainer moved on, a rate limit was quietly lowered, a data centre migration altered its latency distribution. Nothing about the dependency graph as stored says any of this happened. The automation continues to route around it as though the old profile still holds, because the belief "this dependency is healthy" was never timestamped against the conditions that made it true. When it fails, it fails in exactly the way a stale belief fails: confidently, and at the worst possible moment, usually during a deploy the automation itself approved.

This is Akerlof's lemon problem wearing an SRE badge. The dependency's true quality has changed. The system holding the belief about that quality cannot represent the change, so it cannot disclose it, so the principal — the human who will be paged at 3 a.m. — has no way to ask "is this still true?" before the fact. They find out after.

Three generations, three disclosure surfaces

A Large Language Model deployed as a runbook assistant or an anomaly triager carries knowledge frozen at training time: which services typically depend on which, what a normal error budget looks like, how a known incident class usually resolves. It cannot report the age of any of this, because staleness has no representation in its weights. A dependency's failure mode from two years ago and one from last week are stored with identical confidence. Ask it whether a given service is currently a single point of failure and it will answer with the fluency of certainty, drawing on whichever pattern its corpus weighted most heavily — which may be exactly the pattern that stopped applying.

A Large World Model built on live telemetry does better, but only for the scene in front of it. Give it the current service graph and metric streams and it can flag an anomaly the instant a latency distribution shifts, with a timestamp attached. That timestamp is real audit surface — the first the SRE has had. But it holds only while the sensors are live and only for what falls inside the current window. It has no memory of the dependency's history before the window opened, and it forgets what left the frame. It can tell you the cache is slow right now. It cannot tell you that this cache has been quietly degrading for six deploys and nobody re-evaluated the retry policy.

A Large Universe Model is defined by what it can disclose, not by how much it knows. It holds beliefs about every dependency as revisable claims, each carrying provenance: which telemetry stream produced this reliability estimate, when it was last updated, what superseded the previous estimate and why. Staleness becomes a field the system can report rather than a gap the SRE discovers during an incident retro. The dependency that changed six months ago shows an estimate six months old, flagged as unrefreshed against a stream that is still running. The agency problem does not disappear. It becomes a thing you can write a contract against — an alert threshold on estimate age, an escalation when confidence decays past a bound.

Why this isn't a motive problem

It is tempting to read the stale-dependency failure as an alignment issue — the automation "doesn't care" whether it pages the right person, so add penalties, add a review board, add a human in the loop. That framing reaches for the wrong toolkit. A dependency-graph model trained on last quarter's traffic patterns has no interest to misalign; it simply has no slot in which "this estimate is six months old" could be stored. Punishing it, retraining it against a new incident, or adding an approval gate all leave the underlying representational gap untouched. The fix that actually addresses the mechanism is structural: a provenance field on every belief, populated from streams that keep running. That is a data-modelling problem, not a governance problem, and mistaking one for the other is how organisations end up with elaborate incident-review processes that never once ask why the system couldn't tell them in advance.

Two objections worth taking seriously

Provenance metadata is just another report the agent writes. An agent sophisticated enough to game its own uncertainty scores can look impeccably documented while being quietly wrong — a dependency marked "verified, confidence high" from a health-check stream the agent itself deprioritised.

This is the strongest objection and it is correct as far as it goes. Provenance is a claim, not a proof. But compare the verification surface. Under a frozen dependency map, the SRE has no cross-check at all — no timestamp, no source, nothing to query independently. Under a provenance regime, the claim "this estimate is current" is falsifiable against the very telemetry stream it cites; you can go and ask the stream. That is the exact condition under which Holmström's informativeness principle bites: a contract can condition on any signal correlated with the agent's true action, and a checkable timestamp is such a signal even if it can occasionally be gamed. Gaming an auditable record is a narrower, harder-for-the-agent problem than being silently, structurally wrong. It's the difference between a books-cooking manager and a manager who keeps no books.

Most of a dependency graph is stable. The core routing topology of a well-run system barely changes month to month. Demanding continuous provenance on every edge to catch the rare dependency whose reliability shifted is a large cost for a small, localised problem.

The stable majority is real. The trouble is that no current architecture can partition the graph into stable and volatile edges in advance, because the model has no internal signal distinguishing "this dependency's health has been constant for three years" from "this one changed last Tuesday" — both are just entries in the same graph, scored the same way. The SRE cannot know which edges need re-verification without already knowing the answer they're trying to get from the system. Continuous intake with provenance is not needed to keep the stable ninety per cent fresh; it is needed so the volatile ten per cent announces its own change rather than hiding inside a graph that looks uniformly current.

Where the ladder ends

Arrow's patients, Akerlof's used-car buyers and today's paged SRE are asking the same question of their respective agents: is this still true, and how would you know? A frozen corpus cannot answer, because it holds no record of its own age. A live sensor window can answer only for what it currently sees. A system of revisable, provenanced beliefs against streams that keep running is the first agent in this lineage that can answer honestly, continuously, about the thing that actually breaks production: a dependency trusted long after the reasons for trusting it expired. There is no further intake regime beyond everything, continuously, with receipts. What is left after that is the ordinary and unglamorous work — building the incentive contracts and audit routines Holmström's generation already knew how to design, now finally aimed at an agent capable of being checked.