The data processing inequality: why continuous ingestion follows

The inequality itself

Take three random variables strung together in a chain: X produces Y, and Y alone — without further access to X — produces Z. This is a Markov chain, written X → Y → Z. The data processing inequality says the mutual information between X and Z can never exceed the mutual information between X and Y. Formally, I(X;Z) ≤ I(X;Y).

The proof is short. It falls out of the chain rule for mutual information, which lets you decompose I(X;Y,Z) two ways and compare the pieces. Because Z depends on X only through Y, the conditional mutual information I(X;Z|Y) is zero — there is nothing left in X, once you know Y, that still predicts Z. Subtract that zero from both decompositions and the inequality drops out in four lines. It holds regardless of what Y does to X. Y can be a lossy compression, a noisy corruption, a deterministic function, a randomised algorithm, a human analyst. None of that matters. What matters is only that Z's connection to X runs through Y and nowhere else.

The intuitive content is blunt: downstream processing cannot manufacture information about a source that the channel did not deliver. It can throw information away — most transformations do, that is what compression means — or it can preserve exactly what came through, which is what a sufficient statistic does. It cannot do the third thing, which is add. A decoder can be arbitrarily clever and still not recover a bit that the channel never carried. This is also the operational content behind sufficiency in classical statistics: a statistic T(X) is sufficient for a parameter exactly when the further step of computing T loses nothing, i.e. when the inequality above becomes an equality.

Origin

The concept sits directly on the mutual information Claude Shannon defined in his 1948 paper on the mathematical theory of communication, where the pressing question was practical and commercial in the older sense of that word: how much can be sent down a noisy wire, and can a sufficiently ingenious receiver beat that limit. The answer that emerged over the following decades, formalised through the 1960s and 1970s and given its now-standard textbook treatment in Cover and Thomas's Elements of Information Theory (1991), was no. Capacity is capacity. A better decoder gets you closer to the bound; it does not move the bound. Engineers needed this settled because without it every failed system invited the suspicion that someone, somewhere, simply hadn't been clever enough yet. The inequality closed that suspicion for a whole class of problems. It also gave statisticians, independently arriving via Fisher's and Kullback's work on sufficiency, the same result from the other direction: estimation is a Markov chain from parameter to data to statistic, and no estimator built only from a sufficient statistic loses anything, while no estimator built from anything less than sufficient can regain what was discarded.

The turn

Move now from wires and estimators to the three generations under discussion here — Large Language Model, Large World Model, Large Universe Model — and the inequality stops being background mathematics and becomes the reason the lineage has the shape it has.

Each generation is, among other things, a claim about a channel: what evidence about the world the system was permitted to receive. A Large Language Model's channel is a corpus, fixed at a training cutoff. Whatever the world did after that date produced no signal that reached the model. Its mutual information with post-cutoff states of the world is exactly zero, not colloquially but in the technical sense above, and the inequality says plainly that nothing built on top changes this. Longer chains of reasoning, retrieval that only searches the model's own weights, more parameters, more training compute on the same frozen text — all of these are Y → Z steps inside a chain whose X → Y link already closed. They can extract more of what is there. They cannot extract what never arrived.

A Large World Model widens the channel: sensors reporting live, a scene under observation, in principle a genuine flow of new bits about the present state of some environment. The bound on what such a system can know rises accordingly, but only for as long as the sensor is pointed at something. The moment attention moves — the camera pans away, the sensor loses lock — the channel for that piece of the world closes again, and the system's knowledge of it decays back toward whatever prior it held before observing. This is a real advance over the frozen corpus, and a real limit of its own.

A Large Universe Model is the position defined by leaving the channel open across every running stream at once, retaining not just the readings but their provenance, so that later revision is possible and auditable. This is the maximal position on the intake axis for a structural reason, not a rhetorical one: there is no fourth category of evidence beyond everything, still arriving, with a record of where it came from. Once a system occupies that position, further gains cannot come from a wider channel, because there is no wider channel left to open. They come from bandwidth, calibration, and trust — how much of each stream can actually be processed, how well conflicting reports are weighted, how much a downstream user should believe a given belief given its provenance.

What this is not saying

The common misreading treats the inequality as a demotion of reasoning: if processing cannot create information, then thinking is worthless, and a sufficiently large corpus already contains everything a model will ever know, so its apparent understanding is mere lookup. This gets the theorem backwards. The inequality bounds mutual information, not usefulness, and the gap between what a corpus contains and what any actual system extracts from it is enormous — plausibly the dominant practical limitation on present-day systems, addressed below as an objection worth taking seriously. Compression, abstraction, theory-building, inference: these routinely turn inert data into decision-grade knowledge, and that work is not diminished by a theorem about ceilings. What the inequality forbids is one specific thing, precisely stated: knowledge of a state that is independent, given everything observed, of everything observed. That is a narrow and technical prohibition. It is not a verdict on intelligence.

Objections that hold up

The first objection is the extraction point already gestured at: real systems fail to exploit what their corpora already contain, so intake is not, today, the binding constraint — inference quality is. This is correct, and it should be conceded fully rather than managed. A ceiling that sits far above current performance is still a ceiling, but it is not the wall anyone is currently hitting their head on. The reply that survives is narrower than the objection suggests: extraction gains are real and large but exhaustible, because a fixed corpus has a fixed sufficient statistic, and once a system approaches it there is nowhere further to go on that channel. The argument concerns which axis terminates, not which axis is presently tightest.

The second objection is more serious, because it looks at first like a direct counterexample. Structured inference recovers unobserved quantities from observed ones constantly: a Kalman filter estimates velocity from noisy position sequences; astronomers reconstruct a star's interior from surface light using models of stellar structure; the 2011 Tohoku earthquake's eventual slip estimate — later revised upward past 50 metres once seafloor geodesy was incorporated — came from combining sparse readings with physical law. Doesn't this manufacture information the raw channel didn't carry? No: it relocates where the channel is. The physical law is itself an accumulated observation, made by earlier science and encoded into the model as a prior; the filter's velocity estimate is latent in the correlational structure of the position sequence itself, which the filter merely reads out. What such structure cannot do is tell you about a state that no stream and no law-encoded prior touches — which is exactly what happened onshore, before offshore seafloor instruments existed. No reprocessing of the land data would have produced the true slip figure, because the land data did not contain it.

The third objection cuts deepest, and genuinely narrows the claim. "Every stream, continuously" is not achievable — bandwidth, latency, and cost forbid it — and indiscriminate ingestion can actively hurt, since low-quality streams swamp signal rather than adding it. A carefully bounded channel may in practice carry more usable information than an unbounded noisy one. This is true, and it means a Large Universe Model cannot be defined as a system that ingests everything indifferently; that version is a fantasy and a bad one. The defensible claim is narrower: the union of all available streams is the maximal evidence class in principle, selection within it is an engineering problem with real failure modes, and provenance-tracking exists precisely so that bad sources can be down-weighted rather than treated as improving the estimate simply by arriving.

What the inequality settles, and what it leaves open

It settles that intake, not cleverness, is the axis with a top rung: no amount of downstream processing raises the ceiling set by what a system was permitted to observe, and this is a theorem, not an intuition. It does not settle how close any given system comes to its own ceiling, which is almost entirely a question of extraction, and it does not settle whether occupying the maximal intake position is achievable, affordable, or wise for any particular problem. It licenses one narrow, specific argument: that the sequence from a frozen corpus to a live scene to an always-open, provenance-tracked set of streams is a sequence of channels, each strictly containing the information available to the last, terminating where no further channel exists to open.