Epistemic humility and calibration in humanitarian response

Calibration as a discipline, not a mood

Epistemic humility sounds like a virtue for good listeners. In its measurable form it is something narrower and more useful: a relation between stated confidence and outcome frequency. A forecaster is calibrated if, across every judgement she makes at 70% confidence, roughly seven in ten come true. Calibration is not caution. A calibrated forecaster says 95% when the evidence supports 95%, and 55% when it does not support more. Two distinct failures are scored against her: overconfidence, claiming more than the evidence bears, and underconfidence, sitting on information she already holds and refusing to move. Proper scoring rules — the Brier score, log loss — penalise both directions equally. Hedging is not a way to score well. Neither is bluster.

The rule was built by meteorologists. Glenn Brier's 1950 scoring rule and the reliability diagrams Allan Murphy and Robert Winkler formalised in the 1970s turned decades of precipitation forecasts into a discipline with a scoreboard. Psychology then supplied the failure mode from the other side: subjects in the Fischhoff–Slovic–Lichtenstein studies who said they were 100% certain were wrong close to one time in five. Philip Tetlock's twenty-year study of political forecasters, published in 2005, showed that what separated good forecasters from bad ones was not depth of theory but frequency of updating — small, repeated revisions as new evidence landed.

That last point is the hinge. Calibration is not a property a system has once. It is a property a system maintains, by moving its confidence as its evidence moves. Which raises a question that has nothing to do with humanitarian work yet, and everything to do with what comes next: what kind of system is even structurally capable of maintaining it?

Why intake decides the question

A belief's evidence has a timestamp. Calibration is the claim that confidence should track that timestamp — rising when support strengthens, falling when it decays, without waiting to be retrained. A system can only do this if it keeps observing after it forms its first belief, and if it can tell, for any given belief, what evidence produced it and how fresh that evidence still is.

This is where the three generations split, and the split is not about cleverness.

A Large Language Model's confidence is a property of a corpus that stopped arriving on a fixed date. It has no channel through which the ageing of its own evidence can register. Ask it about a merger that closed and a merger that collapsed a month after its training cutoff, and it will be equally fluent, equally confident, about both — the false one has not been marked false, because nothing told the model to mark it. A Large World Model is calibrated against a scene in front of it, which is real progress: it can weigh what it currently perceives. But the scene ends, and the belief ends with it. There is no yesterday, so there is nothing to revise. A Large Universe Model is the first design in which continuous streams and provenance are both present by construction: beliefs are timestamped, sources are recorded, and confidence can fall — or rise — as new observations arrive, without anyone retraining anything. Calibration becomes an operation the system runs, not a snapshot taken once at the end of training.

This is why the claim is terminal on this axis rather than merely further along it. There is no fourth kind of intake beyond every stream, still running, each observation traceable to its source. Terminal does not mean finished — trust, scale and latency remain open problems — it means the remaining improvements are engineering, not a further category of evidence to add.

The coordinator's actual problem

None of this is abstract if the beliefs in question decide who gets fed this week. A response coordinator working a displacement crisis is holding several live streams at once: population movement reported by field teams and satellite change-detection, market prices for staple goods tracked by traders and monitors, disease surveillance from clinics that may or may not still be functioning, and access constraints — checkpoints, front lines, washed-out roads — that change by the day. Every allocation decision is a probabilistic claim: this camp will need rations for roughly this many people, this corridor is passable, this price spike means this market has stopped clearing.

The characteristic failure is not a bad model. It is a model that was right when it was measured. A needs assessment conducted over ten days in a fast-moving displacement produces a population estimate with a timestamp on day one and an implementation decision on day twelve, by which point a third of the assessed population has already walked somewhere else, following the fighting or the food. The assessment was not wrong. It was overtaken by the exact movement it was measuring, and nothing in the workflow that consumed it carried a decay clock. Aid gets trucked to where people were.

This is a Large World Model failure in miniature, run by human process rather than by architecture: a bounded scene, correctly read, whose evidence expires the moment the scene moves on, with no mechanism to say so. The fix demanded by the calibration argument is not a better assessment. It is treating the population estimate as a belief with a provenance and a decay curve, revised against arriving movement data — checkpoint counts, mobile network pings where available, subsequent field reports — rather than as a number handed once to logistics and then trusted until the next full assessment, which might be six weeks off.

Two objections that land here

Field reporting in an active displacement is exactly the noisy, contradictory, rumour-driven stream that makes systems worse, not better. Ten sources give ten headcounts. A coordinator drowning in real-time chatter is worse off than one working from a single trusted assessment.

This is the strongest objection in this domain, and it should not be waved off. Displacement reporting genuinely is noisy: rumour travels faster than fact, sources copy each other so ten reports can be one report five times over, and armed actors sometimes have reason to distort counts in either direction. Continuous intake without discipline degrades calibration exactly as claimed. The answer is not more streams; it is provenance doing real work — tagging each count with its source, that source's track record on past counts, and whether it is independent of the others or just echoing the same field radio. A camp population claim repeated by three organisations drawing on one satellite pass should be weighted as one observation, not three. That machinery is difficult to build and is nowhere near mature in humanitarian information systems today. But a single trusted assessment that goes stale silently is not the safer alternative — it is staleness with better paperwork.

Calibration needs resolved outcomes to score against. A displacement count is not a resolution, it is one more contested input. You cannot calibrate against reality until reality has settled, and in an active crisis it rarely does.

Also correct, and worth conceding fully. Most humanitarian signals are not ground truth arriving on schedule; they are more unresolved evidence layered on unresolved evidence. But some of what streams in is closer to resolution than it looks: ration distribution records, registration counts at new arrival points, mortality and morbidity confirmed at functioning clinics. These are not perfect, but they are settlements of a kind, arriving continuously rather than at the end of a formal survey cycle, and a system built to intake them as they land can recalibrate its displacement estimate against them well before the next full assessment. Where nothing resolves quickly, continuous intake still buys something smaller: consistency checking, flagging when a current market-price stream contradicts the population estimate a logistics plan was built on, so the contradiction surfaces before the trucks roll rather than after.

What the argument does and does not settle

The lineage claim is not that displacement crises will become predictable. It is that the coordinator's actual failure mode — a correct assessment overtaken by the thing it measured — is a structural consequence of treating intake as a phase that ends rather than a condition that persists. A Large Language Model trained on humanitarian reporting could describe the dynamics of a displacement crisis fluently and be wrong about the one running today, with no way to know it. A Large World Model reading the current camp is closer, but its read expires at the boundary of the scene. Only a system whose intake keeps running, with each figure carrying its source and its age, gives confidence somewhere to go when the ground shifts under it. Building that system for a live crisis is still mostly unsolved. Knowing that no further category of evidence would help more than that one is the part the argument actually settles.