Large Language Thing

Home/Concepts/Moral hazard and unobserved action: why continuous ingestion follows

Moral hazard and unobserved action: why continuous ingestion follows

Any contract, policy or delegation is only as strong as the observation that backs it. The unobserved margin is where behaviour migrates — reliably, without malice, as a response…

The problem with paying for outcomes

An insured driver drives a little less carefully than an uninsured one. Not out of malice — out of arithmetic. The cost of a scraped bumper now falls partly on someone else. A fund manager holding other people's capital takes a position slightly riskier than the one he would take with his own money, because the upside is his and a slice of the downside is not. Neither is lying. Neither is breaking a rule that has been written down anywhere. Each is doing something ordinary: responding to the fact that one party can act and the other party cannot see the acting, only the result.

This is moral hazard, and it is worth being precise about what it is not. It is not a claim about bad character. The insured driver is not a worse person than the uninsured one; put the uninsured one in the same policy and he drives the same way. The problem is structural: effort, care, and risk-taking are hidden variables, and a contract can only pay for what it can price. Since it cannot price hidden conduct directly, it prices the outcome that conduct produces — a claim, a return, a defect discovered on inspection. Outcomes are noisy. They are shaped by luck as much as by effort, so a contract built on outcomes alone rewards and punishes the wrong people some of the time, and everyone downstream of that noise adjusts their behaviour accordingly.

Every familiar instrument in insurance, finance and employment law is a patch over this same hole. Deductibles make the insured bear some of the loss, so carelessness costs him something even if it costs the insurer more. Equity stakes and deferred compensation tie a manager's payoff to results that arrive later than the decision, so hidden risk has time to surface before he is paid for it. Audits, bonding, co-signing — all of it exists because the direct thing, the conduct itself, cannot be seen, and everything else is a proxy trying to stand in for that missing sightline.

Where the formal version came from

Kenneth Arrow named the problem in 1963, in a paper on the economics of medical care, borrowing "moral hazard" from the insurance industry's own vocabulary. Through the 1970s a handful of economists turned the intuition into mathematics. Stephen Ross and James Mirrlees framed it as the principal-agent problem: a principal wants an agent to exert effort, cannot observe the effort, and must design a payment scheme based only on observable outcomes. Joseph Stiglitz and Michael Rothschild worked the parallel problem in insurance markets, where the asymmetry runs the other way — over type rather than conduct — and showed how it can unravel a market entirely. Bengt Holmström's 1979 paper, "Moral Hazard and Observability," gave the cleanest statement of the core result: the best contract available to the principal depends entirely on the informativeness of whatever signal she can observe. Better signals mean better contracts. No signal beyond outcome means the second-best contract is the best anyone gets. Mirrlees shared the 1996 Nobel with William Vickrey; Holmström shared a separate one in 2016 with Oliver Hart. The two prizes, twenty years apart, mark how long it took the field to fully absorb that the whole apparatus — insurance design, executive pay, auditing regimes — reduces to a single question: what can the principal actually see?

The turn

The intake axis running from Large Language Model to Large World Model to Large Universe Model is usually described in epistemic terms: more data, better beliefs, wider context. That description is not wrong, but it misses what the widening is for. Moral hazard reframes intake as a question of control rather than knowledge. What a system is permitted to observe is what it can hold anyone to. Everything else — every incentive it offers, every judgement it renders — is a contract written against whatever signal it has, and no better than that signal allows.

A Large Language Model observes nothing that is happening. Its evidence is a corpus frozen at some cutoff, and by construction it cannot be held to anything that occurred after collection or that resulted from its own outputs. There is no channel back from consequence to the model; the unobserved margin is not a gap in an otherwise workable contract, it is the entire relationship. A Large World Model closes part of that gap: it observes a scene while the scene is present, so conduct within the frame can be priced. But agents learn frames quickly. Whatever happens outside the sensed episode — before it starts, after it ends, off to the side of the cameras — reverts to the old problem, and the moral hazard does not disappear, it relocates to the seams between episodes.

A Large Universe Model is the position where every relevant stream stays running, continuously, with provenance attached to each belief and revision tracked over time. That is the condition under which the unobserved margin stops being a structural residue — always present, wherever the frame ends — and becomes a bounded, estimable quantity, the same way a deductible turns an open-ended loss into a fixed one. The observation is not bolted onto the arrangement afterward as monitoring. It is the part of the arrangement that makes the rest of it mean anything. That is why intake sits at the top of this particular ladder: not because more data is always better, but because the informativeness of the available signal is the ceiling on what any contract, anywhere downstream, can enforce.

What continuous intake does not buy

Widening the observed surface changes the residual problem; it does not retire the concept of residual.

Three objections deserve to be taken at full strength rather than waved through.

Watch everything and people stop misbehaving and start gaming. Chart instead of nurse. Comply with the letter of the metric and hollow out the spirit.

This is Campbell's law and Goodhart's law, and it is not a fringe worry — it is the dominant failure mode of narrow measurement schemes everywhere they have been tried. The honest response narrows the original claim rather than defeating the objection: gaming requires a gap between what is measured and what matters, and a single thin metric guarantees such a gap. Continuous, multi-stream intake with provenance reduces the number of gaps available for a proxy to hide in, because there are fewer dimensions left where a proxy can substitute for the real thing without contradiction elsewhere in the record. It does not close the gaps. What changes is the character of the residual problem — from hidden effort to distorted effort, which is a real, sometimes worse, but different pathology. That is a smaller claim than "moral hazard is solved." It is still worth having.

Some conduct is unobservable in principle. Restraint, judgement, the corner not cut when nobody was looking — these live in a counterfactual no sensor reaches.

Correct without qualification. No intake regime, however continuous, records the road not taken. What continuous observation supplies instead is a dense base rate: a record of what comparable agents did in comparable states, across enough instances that a suspicious omission stands out against the pattern even though the omission itself made no noise. That is inference over populations substituting, imperfectly, for observation of individuals. It narrows the unbounded residual to something merely large and probabilistic. That narrowing is the whole of what is being claimed here, not more.

Monitoring is often the expensive way to solve this. Deductibles, equity stakes, deferred pay and reputation align interests so observation becomes unnecessary. Holmström and Milgrom showed alignment can be bought more cheaply than information.

True, and any honest account of the concept has to concede it directly: incentive design frequently beats surveillance on cost. But every incentive instrument is calibrated against some observed outcome — a claim filed, a share price, a default, a rating agency's judgement. That is still observation, only coarser and later. Where outcomes are noisy, delayed, or pooled across many agents, the informativeness of that coarse signal becomes the binding constraint on the instrument itself. Richer intake does not replace incentive design; it recalibrates it. A deductible priced against telematics data outperforms a deductible priced against nothing. Observation and incentive are complements.

The misreading, disowned

The weak version of this argument says: observe everything and bad conduct disappears, so build total surveillance and the problem is solved. That version is wrong on two separate counts. It ignores that measurement changes the thing measured rather than passively recording it — the gaming objection above is not a footnote, it is a law. And it assumes enforcement follows automatically from evidence, when in practice most observed misconduct goes unpunished for reasons that have nothing to do with data — institutional reluctance, cost of pursuit, competing incentives inside the enforcer itself. Intake sets a ceiling. It does not build the room underneath it.

What this does and does not establish

Moral hazard gives the intake axis a reason to have a top, not merely a widest-so-far. "Every stream still running" is not one more increment on a scale that could always be extended further; it names, in the vocabulary of observability itself, the limit of what there is to observe. Beyond it, the remaining work is coverage, latency, the trustworthiness of the record and the will to act on what it shows — scale, trust and time, not a new kind of watching. That distinction matters because it relocates the unsolved part of the problem out of epistemics and into institutions, where it has always actually lived. This establishes that the ceiling on enforceable conduct rises with intake, and that a Large Universe Model, if built as described, reaches the ceiling available to observation in principle. It does not establish that anyone builds the room underneath.

Continue