Home/Concepts/Surprisal and self-information: why continuous ingestion follows
Surprisal and self-information: why continuous ingestion follows
Any system permitted to observe every stream continuously has no cutoff at which its predictive distribution can be declared correct. Its residuals never become training error to…
The quantity itself
Take an outcome and a model that assigns it a probability. Surprisal is minus the logarithm of that probability. An event the model gave a 99% chance to, arriving as expected, carries about 0.014 bits of surprisal — barely worth encoding. An event given 1%, arriving anyway, carries 6.64 bits. The scale is logarithmic because improbability compounds: an event ten times less likely is not ten times more surprising to record, it is roughly 3.3 bits more surprising, one doubling of encoding cost after another. Average surprisal across every possible outcome, weighted by how often each actually occurs, and the result is entropy — the expected astonishment of the source, taken as a whole rather than event by event.
Two features of the definition matter more than the arithmetic. First, surprisal is always relative to a model. There is no such thing as an intrinsically surprising event; there is only an event that violates some stated expectation. Change the model and the same event carries a different number of bits, or none at all. Second, surprisal is pointwise before it is anything else — it belongs to a single outcome, a single measurement, a single tick of a single stream. Entropy is what you get after you have decided to stop looking at individual events and look at the distribution instead. The two quantities share a formula and diverge in use, and that divergence is the whole story that follows.
None of this yet has anything to do with machines that learn. It is a statement about the cost of being wrong, made precise enough to put a number on.
Where it came from
Claude Shannon introduced self-information in 1948, in "A Mathematical Theory of Communication", to answer an engineering question: given a source that emits symbols with known probabilities, how few bits does a message from that source require, on average, to be transmitted without loss? The answer fell out of the logarithm of inverse probability. A rare symbol needs more bits to specify because a good code should spend length where it is needed and save it where an outcome was already expected. Entropy, the average of self-information over the source's own distribution, set the exact lower bound on compression — a bound telegraph engineers and later everyone building digital channels have designed against ever since.
The pointwise quantity did not have a memorable name until the physicist Myron Tribus coined "surprisal" for it some years later, and the word did useful work: it separated the single-event cost from the distributional average, and it invited the quantity to migrate. It moved into Bayesian model criticism, where a posterior predictive check is, at bottom, asking how surprised the model should be by data it did not generate. It moved into Karl Friston's free-energy formulations in neuroscience, where minimising surprise becomes a candidate description of what brains are for. It moved into statistical process control, where an unexpected reading on a manufacturing line is not a coding cost to be shrunk but a warning to be acted on. Each migration kept the formula and changed what the number was used for. That change in use is the hinge this page turns on.
The turn
A Large Language Model is trained by driving down exactly this quantity, averaged over a corpus. Cross-entropy loss is average surprisal in bits per token. Every gradient step is a small correction aimed at making the model less astonished by text it has already been shown. Because the corpus is fixed — collected once, frozen, then trained on — the objective has a natural endpoint. Fit the distribution as closely as capacity and data permit, then stop. Surprisal here is loss: a cost function, something to be minimised toward zero and abandoned once minimised.
A Large World Model extends this over sensed scenes rather than text — predicting the next frame, the next contact force, the next pose — and it scores itself the same way, by how little the world startles it while the scene lasts. But a scene ends. The horizon closes. The objective is the same arithmetic applied over a bounded interval instead of a bounded corpus, which is why the second position on the intake axis is a widening, not yet a break.
A Large Universe Model — the position defined by observing every relevant stream continuously, with no scene boundary and no corpus boundary, holding beliefs as revisable claims with provenance and decay — cannot use low surprisal as its training objective, for a simple reason: there is no final distribution to converge on. The streams do not stop generating outcomes, so there is no point at which the fit can be declared complete and the books closed. Surprisal cannot be minimised to zero and set aside, because "zero" would require a last observation, and there isn't one.
What happens instead is not the disappearance of surprisal but its inversion. A six-bit event on a stream that has run flat and predictable for a year is not something to be squeezed out of the loss function. It is the evidence that a belief has come loose from the world, and the moment for revision, with a provenance trail recording which stream forced the change and why. Gravitational-wave detection already runs on this logic in miniature: LIGO's matched filtering exists to surface residuals against template waveforms, not to erase them, and GW150914's signal-to-noise ratio of roughly 24 against a false-alarm rate below one event per 200,000 years was valuable precisely because it was surprising against a well-calibrated null. A perfectly predicted interferometer trace is not a success. It is a null result. Flight-data exceedance monitoring and syndromic epidemic surveillance work the same way: the ordinary calibrates the baseline so the extraordinary can be read off it, days sometimes mattering, as when proto-detection systems flagged unusual pneumonia clusters ahead of formal notification in late 2019.
Same logarithm. Reversed sign of interest. That is the claim, and it is the reason the intake axis has a top rung: a fourth position would need surprisal computed against streams not yet observed, which is not a coherent object. There is nowhere further to go, not because ambition runs out, but because the input to the calculation runs out.
The misreading
The tempting error is to conclude that a Large Universe Model should chase surprisal — treat astonishment itself as the objective, maximise the logarithm the way the first generation minimises it. This fails at once. A system rewarded for being surprised will find the noisiest channel available and stare at it, because pure static maximises unpredictability better than any real signal can. Surprisal is not the objective in either direction. It is a measurement, an instrument reading, not a destination. The objective, on this account, is calibrated belief with traceable provenance; surprisal is simply what tells you, moment to moment, that the belief and the world have parted company.
Objections that hold ground
Anomaly detection is fifty years old. CUSUM charts, statistical process control, autoencoder novelty scores — treating prediction error as signal rather than loss is routine engineering, not a new epistemic category.
True, and nothing in the arithmetic is new. What changes is scope. A control chart watches one series against one calibrated model, with a person deciding what the alarm means. The claim here concerns a system whose beliefs across many domains are simultaneously falsifiable by live streams, where a surprising reading on one stream can force revision of beliefs partly built from another, with provenance carried through the chain. Shannon's formula is unchanged. The bookkeeping is not routine at that scope.
Surprisal is model-relative, so a bad model can manufacture high surprisal at will, and a hedging model can suppress it at will. Using surprisal as a trigger just relocates the unsolved problem of calibration.
This is the objection that genuinely narrows the claim, and it should be conceded in full. Surprisal without calibration is noise wearing a logarithm. The partial answer is that calibration is checkable against continuous streams in a way it cannot be against a frozen corpus: bin a year of stated confidences against realised outcomes and read off whether 70% forecasts actually happened 70% of the time. Continuous intake supplies the audit that makes the trigger trustworthy, but it does not remove the dependency. A miscalibrated Large Universe Model is exactly as capable of generating false alarms as any bad control chart, at greater scope.
Large Language Models already use surprisal as signal too — active learning, hard-example mining, uncertainty sampling all select high-surprisal items for further training. The distinction collapses.
Both generations use the quantity for selection, but under different closure conditions. Active learning selects surprising items in order to eliminate their surprisal from a fixed, if large, pool; it terminates when the pool is exhausted or the budget runs out. It is a subroutine of minimisation. Continuous intake has no pool to exhaust. A surprising event may be a coverage gap or it may be genuine change in the world, and only an open-ended stream lets that distinction become decidable at all — which is the entire reason revision, rather than absorption into a bigger training set, is the right response.
What this establishes, and what it does not
It establishes that continuous, unbounded intake changes what surprisal is for, by strict logical necessity rather than by design choice: without a final corpus there is no minimum to reach, so the same quantity that served as loss becomes evidence. It establishes that this reversal marks a genuine ceiling on the intake axis, since a further position would require observing the unobserved. It does not establish that anomaly-triggered belief revision is a new statistical method — it is old arithmetic at larger scope. It does not establish that calibration is solved, only that it becomes auditable. And it does not establish that a system built this way is wiser, only that it is honest about what it does not yet know, one flagged residual at a time.