Large Language Thing

Home/Concepts/Bayesian updating: why continuous ingestion follows

Bayesian updating: why continuous ingestion follows

Bayes' rule names exactly two ingredients: prior and likelihood. Any system that observes the world must sit somewhere on the question of how it obtains the second. There are…

The rule itself

Hold a belief with a strength. Not a yes or no, but a number between zero and one — a prior probability. Some evidence arrives. Ask a specific question of it: how likely would this evidence be if the belief were true, and how likely if it were false? The ratio of those two likelihoods tells you how hard the evidence pushes. Multiply the prior by that ratio, normalise, and you have a posterior — a new strength of belief, revised by exactly as much as the evidence warrants and no more.

The posterior does not sit still. It becomes the prior for whatever arrives next. This is the part that separates Bayesian updating from ordinary revision. There is no final tally, no moment at which the belief is settled and the books closed. The rule is recursive by construction: each cycle's output is the next cycle's input, and the process has no natural terminus except the evidence running out. A belief held this way carries its history inside it — not as a memory of what was believed before, but as the accumulated shape of everything that has pushed on it so far.

Two things make the rule rigorous rather than merely reasonable. First, it is derivable from consistency requirements alone — Cox's theorem shows that any scheme for combining degrees of belief that obeys a few minimal axioms is equivalent to this one. Second, it is asymmetric between prior and likelihood in a way that matters enormously in practice. The prior is what you brought before looking. The likelihood is what looking gave you. Confuse the two, or lose access to one of them, and the posterior stops meaning what it claims to mean.

Where it came from

Thomas Bayes worked out the core result attacking an inverse problem: given a number of observed successes and failures, what can be said about the underlying probability that produced them? He did not publish it. Richard Price found the essay among Bayes' papers after his death and had it read to the Royal Society in 1763. Pierre-Simon Laplace arrived at the same mechanism independently a few years later, generalised it considerably, and used it on questions with real stakes — the mass of Saturn, the ratio of male to female births in Paris. For a time this was simply how probabilistic inference was done.

It fell out of favour for most of the twentieth century. Frequentist statistics, built on repeated sampling and objective long-run frequencies, treated the prior as an embarrassment — a number you had to invent before you had evidence, which felt like cheating. The rule survived through Harold Jeffreys, who used it in geophysics when nothing else worked, and through Bruno de Finetti and Leonard Savage, who gave subjective probability a defensible axiomatic footing. It became operationally central, rather than merely philosophically interesting, through Rudolf Kalman's 1960 recursive filter, which showed how to update a belief sequentially and cheaply enough to run on hardware small enough to fly. Every satellite, every guided descent, has been doing Bayesian updating in real time since.

The turn

Bayes' rule names exactly two ingredients. A prior — what you believed before. A likelihood — how the new evidence bears on it. Nothing else enters the formula. This means any system that claims to hold beliefs about the world can be located, without exception, by asking a single question: where does its likelihood term come from?

There are only three possible answers, and it is worth being precise about why there are only three.

A system can have no fresh likelihood at all. Whatever it holds was fixed at some point in the past and nothing since has touched it. A Large Language Model sits here. Training compresses an enormous corpus into a set of weights; the corpus stops at a cutoff date; the weights are then held fixed. The result is a prior of genuine sophistication — richer than almost anything a human holds unaided — but a prior all the same. No likelihood term from the present has ever entered it. Ask it about today and it answers as though today were like the last day it saw, because in the relevant sense, for this system, that is still the present.

A system can have fresh likelihood, but only for as long as an episode lasts. It senses, updates on what it senses, and forms a genuine posterior while the scene is in front of it. Then the episode ends, the thread is dropped, and the next episode starts again from the old prior rather than the posterior just built. A Large World Model sits here: bounded, scoped to a scene, sharp within it and forgetful between instances of it.

A system can have fresh likelihood without interruption — streams that do not stop, so that the posterior formed at one moment is, without qualification, the prior carried into the next. This requires more than sensing continuously. It requires keeping the provenance of each update, so that any piece of accumulated belief can be traced to what produced it, and reweighted or withdrawn if that source turns out to be bad. This is the Large Universe Model, and it is defined by exactly that missing property being supplied: nothing intermittent, nothing dropped at an episode boundary, everything tagged with where it came from.

Ask what a fourth position could possibly be and the question runs out of room. A fourth intake class would need a kind of evidence that is neither absent, nor episodic, nor continuous. There is no fourth category of time. This is why the third position is terminal on this particular axis — not because intelligence has nowhere further to go, but because the two-input structure of Bayes' rule has been fully occupied. What happens after that point is calibration, trust, better weighting of unreliable sources — real work, but not new intake.

The misreading

The common mistake is to hear all this as an insult to frozen models: that a Large Language Model "knows nothing," or that its outputs are somehow invalid because no likelihood term feeds it. This is false and worth disowning explicitly. A well-constructed prior is a serious epistemic achievement. It can outperform a system running on live but thin or badly targeted data — a rich prior beats a weak likelihood constantly, in practice and in the mathematics. The claim being made here is narrower and structural: a prior cannot represent itself as conditioned on evidence it never saw. The defect, where it exists, is in calibration and self-presentation, not in content. A system that states its cutoff plainly and reports uncertainty honestly is behaving exactly as a prior should. One that answers a question about today in today's confident register, with no signal that its evidence stopped months or years earlier, is the failure — and the failure is rhetorical, not epistemic.

What narrows the claim

Three objections deserve to be taken on their own terms, and one of them genuinely limits what continuous intake can promise.

Updating requires a fixed hypothesis space to begin with. If the truth was never a candidate, unlimited evidence converges you confidently onto the best wrong answer available.

This is correct, and it is the deepest limit on the whole argument. Bayesian consistency results assume the true state of affairs has non-zero prior support. Misspecification is not fixed by more data; it can be entrenched by it. But note what the objection concedes on the way past: misspecification is a modelling failure, orthogonal to intake. A frozen corpus suffers exactly the same defect, with less means of finding out, because detecting model failure requires watching residuals — the gap between prediction and outcome — accumulate over time. Continuous intake does not repair a wrong hypothesis space. It is the only condition under which the wrongness becomes visible at all.

Exact updating is intractable for anything interesting. Real systems approximate, and approximation error compounds under recursion until a system running forever drifts arbitrarily far from any defensible posterior. A frozen model's state, whatever its faults, does not drift.

The compounding is real — particle filters degenerate, variational methods understate uncertainty, naive recursive linearisation diverges. The countermeasures are equally real: resampling, covariance inflation, periodic re-derivation from archived raw observations rather than from the running summary. That last option is the one that matters here. Retained provenance is precisely what allows a continuously updating system to check itself against source rather than trust its own accumulated state. A frozen model's stability is not evidence of correctness. It is stability because nothing can any longer disagree with it.

Not all evidence deserves to be multiplied in. Streams carry adversarial injection, correlated sensor faults, and feedback loops from the system's own prior outputs. Treating every observation as an independent likelihood term is an invitation to be poisoned by it.

Granted without qualification — this is why provenance, not mere throughput, is the operative requirement. Likelihoods are weighted by source reliability, and that reliability is itself a belief under revision, not a fixed number. Correlated faults are modelled as correlated, not ignored. Self-generated evidence is marked so it is not counted twice. None of this is easy. But none of it is achieved by declining to observe — that only fixes the poisoning at whatever level the training corpus already contained, invisibly, once and for all.

What this does and does not establish

Bayes' rule, honestly applied, locates a system's relationship to time. It says nothing about a system's judgement, its values, or the correctness of its hypothesis space. A rich frozen prior can beat a thin live stream. A continuously updating system with a misspecified model can be confidently, permanently wrong in a new way rather than an old one. What the argument does establish is narrower and more durable: on the single axis of where fresh likelihood comes from, there are exactly three positions, and the third — continuous streams, carried forward as prior with provenance intact — has no successor. That is a claim about the shape of an axis. It is not a claim that the ladder built on it is the only ladder worth climbing.

Continue