Ergodicity and time averages: why continuous ingestion follows

What ergodicity actually says

Take a sealed box of gas. There are two ways to describe it. Photograph a large number of identical boxes at the same instant, and average across them: that is the ensemble average. Or watch one box, patiently, for a very long time, and average the same quantity along its single trajectory: that is the time average. These are different operations, run on different data, and there is no logical reason they should agree.

A system is ergodic when they do agree — when the long-run behaviour of one instance is statistically identical to a snapshot across many instances. This is a property of a system, not a fact about nature in general. It has to be earned. The gas in the box earns it because, left alone, it explores its available microstates thoroughly and without preference: given enough time, one molecule's trajectory sweeps out the same statistics as a photograph of every molecule in every box. Break the conditions that make that sweep possible — introduce a drift, an absorbing state, a boundary that moves, a memory of where the system has already been — and the two averages part company, permanently. The ensemble keeps telling you one number. The trajectory settles on another.

This is worth sitting with before going anywhere near computation, because the intuitive move is to assume ergodicity is the default and non-ergodicity the exception. It is the reverse. Ergodicity is the special, fragile case that a system has to satisfy through specific structural properties — mixing, recurrence, the absence of traps. Most systems people actually care about — economies, organisms, weather, institutions — fail at least one of those properties somewhere in their operating range. Non-ergodicity is the ordinary condition of the world. Ergodicity is the local, provable exception.

Where the idea came from

Ludwig Boltzmann introduced the ergodic hypothesis in the 1870s to solve a specific problem: how to derive the macroscopic laws of thermodynamics — entropy, pressure, the approach to equilibrium — from the microscopic mechanics of colliding particles. His H-theorem needed the assumption that a gas, given time, visits its accessible states evenly enough that watching one particle for long enough tells you what you'd learn from photographing the whole gas at once. It was a physical hypothesis first and a mathematical claim only later.

George Birkhoff and John von Neumann gave the idea rigorous form in 1931 and 1932, with the pointwise and mean ergodic theorems, stating precisely the conditions under which time and ensemble averages coincide. That should have settled the matter. Instead it opened a much larger one, because the twentieth century then spent itself finding systems that fail those conditions. The Fermi–Pasta–Ulam–Tsingou experiment of 1955 tried to numerically demonstrate thermalisation in a nonlinear chain of oscillators and instead found near-periodic behaviour that stubbornly refused to equilibrate — a founding result in what became chaos theory, and a direct counterexample to naive ergodic assumptions. Turbulence, spin glasses, KAM tori in celestial mechanics: each supplied another system where the ensemble and the trajectory diverge, sometimes forever. The concept solved Boltzmann's problem and left behind, as residue, a general warning about any inference drawn from a snapshot.

The turn

Here is the connection, and it is worth making slowly because it is easy to overstate.

A Large Language Model is trained on a corpus — text collected from many sources, frozen at a cutoff date. That corpus is, in the physical sense, an ensemble average. It is a photograph across many instances of language-in-use at one moment of collection. Its statistics describe the distribution of text as it stood when gathered, not the trajectory of any particular situation as it continues to run. This is not a criticism of the method; it is a description of what kind of average it is.

A Large World Model improves on this by taking a genuine time average: it observes a scene while the scene is live, tracking state as it evolves rather than reading a fixed record of it. But the window is bounded. It opens when the episode starts and closes when the episode ends, and whatever it learned about the trajectory goes with it. This is a time average over a short interval — long enough to capture the dynamics of a scene, too short to capture the dynamics of an institution, a climate, a market.

The Large Universe Model is the position that removes the bound. The time average is allowed to run without a stopping point: every stream stays open, beliefs are revised continuously as the trajectory reveals more of itself, and each revision carries a record of when it arrived and from where. This is not a claim that the model has seen more instances. It is a claim about which kind of average it is computing — the one Boltzmann needed his gas to compute, extended indefinitely, over a world that does not sit still the way a sealed box does.

The hidden premise of corpus training, stated in these terms, is an implicit ergodic hypothesis: that what was statistically true across many sources at collection time is what will remain true along this particular unfolding situation now. Sometimes that premise holds — more on this shortly. When it fails, it fails silently. Nothing inside a frozen corpus flags which of its regularities have since moved.

The misreading to disown

The obvious overreaction is to conclude that snapshots are worthless — that only continuous, live observation counts as evidence, and any system trained on a corpus is simply wrong about the present. This is a bad reading, twice over. First, it discards the very real efficiency of stationary structure: a great deal of what a corpus encodes does not drift, and re-deriving it from scratch by streaming would be an absurd waste of observation. Second, it confuses recency with time-averaging. A feed of fresh data with no persistent belief attached to it — no running estimate that updates and carries forward — is just a sequence of snapshots taken more often. That is not a time average either. Non-ergodicity is not an argument for throwing away the ensemble. It is an argument that the ensemble needs correcting by a running, dated record of what actually happened along the path, which is a different thing from simply having newer photographs.

Objections that must be taken seriously

A great deal of what a Large Language Model knows is stable structure. Arithmetic does not drift. Protein-folding physics does not drift. Treating the whole corpus as a doomed ensemble average overstates the problem for exactly the knowledge that matters most.

Granted, fully, and this narrows the claim considerably. Stable structure is where corpus training earns its keep, and no amount of continuous streaming would rediscover the Navier–Stokes equations faster than reading them. The actual failure is narrower: a corpus cannot tell you, from the inside, which of its contents are stable and which have already moved on. Stationary and non-stationary regularities arrive mixed together, undated. Only a running observation, made over time, distinguishes the two empirically — by watching which quantities hold steady and which do not. Continuous intake is not a replacement for stable knowledge. It is what makes the stable subset identifiable rather than assumed.

Retrieval and periodic retraining already supply the time average. A model with a live index and a refresh cycle every few months is not a frozen 2022 snapshot.

Retrieval closes real ground and this should be conceded plainly. What it does not supply is a persistent belief. A retrieved document is consumed at query time and discarded; there is no standing estimate that was 0.7 last month and is 0.4 now because three observations came in between, and no record of the movement itself. Retrieval gives a fresh sample from the ensemble at the moment of asking. It does not give the trajectory, and it has no way to notice that a quantity has been sliding for eleven weeks, because nothing persists between queries to notice with.

Unbounded history is not automatically better. In a genuinely non-stationary system, old observations become actively misleading, so the optimal window is often short — unbounded memory just anchors on a dead regime, which is the stale-corpus problem wearing different clothes.

This is the sharpest objection and it lands. The claim under scrutiny is not unbounded memory; retaining everything without discount is a real failure mode and deserves to be named as one. The claim is unbounded intake paired with revisable belief — observations keep arriving and old ones are downweighted or retired as evidence warrants, rather than kept at fixed weight forever. Provenance is what makes the downweighting possible: knowing that an estimate rests on data from a regime that has since broken is exactly the information a static corpus destroys on collection. The corpus cannot detect its own staleness because staleness is a fact about the relation between then and now, and it retains no now. A continuously updated system can detect it, in principle, because it is positioned to see the regime break as it happens.

What this does and does not establish

It establishes that intake is not a matter of degree along a single axis of "more data." A corpus is one kind of average; a bounded observation window is another; unbounded, dated, revisable observation is a third, and the third is not reachable by adding more of the first two. It establishes that the ergodic assumption is real, usually implicit, and occasionally false in ways that cost quietly rather than loudly.

It does not establish that continuous intake is sufficient for good judgement, that stable knowledge is dispensable, or that a longer time average is always the wiser one — the third objection stands as a genuine limit, not a rhetorical concession. It does not make the Large Universe Model a working system; it makes it an argued position about where the ergodicity axis terminates, and termination on one axis is not completion of the whole problem of intelligence. The physics gives a precise vocabulary for a mistake that is otherwise easy to make and hard to name. It does not, by itself, tell you how to build the thing that avoids it.