Write-ahead logging and the event log: why continuous ingestion follows

The write-ahead rule

A database that changes a record in place destroys the evidence of what the record used to contain. The old value is gone the instant the new one lands; there is no way to ask what was there a moment before, only what is there now. This is fine until something interrupts the write halfway through — a power failure, a crashed process, a torn page — and the database is left holding a record that is neither the old value nor the new one, but a corrupted hybrid of both.

Write-ahead logging solves this by inverting the order of operations. Before a page in the database is touched, the intended change is appended to a separate, durable, strictly sequential log. Only after the log entry is safely written does the actual page get modified. If the system crashes mid-update, recovery does not try to inspect the half-written page and guess. It reads the log and replays it: redo the changes that were logged but not yet applied, undo the changes that were logged as part of a transaction that never committed. The log is authoritative. The page is a cache of what the log implies.

Pushed one step further, this stops being a recovery mechanism and becomes a description of what state is. Current state is not a fundamental thing that occasionally gets backed up by a log. Current state is a fold — an accumulation — over an ordered sequence of immutable events. The page, the record, the row you actually query: that is a derived, disposable, recomputable projection. The sequence that produced it is the thing that persists. If the derivation is wrong, or a later fact contradicts an earlier one, or you need to know what was believed before some event arrived, none of that requires touching the projection. You touch the log, and you replay.

Where it came from

The technique was formalised in the database systems work at IBM in the 1970s and 80s, built around System R and its descendants, in response to a narrow and unglamorous problem: crash consistency. A machine that loses power in the middle of updating a bank ledger must not leave that ledger in an inconsistent state, half-credited, half-debited. C. Mohan's ARIES papers, published around 1992, fixed the canonical protocol — log before page, redo followed by undo, recovery expressed as logical operations on physical pages rather than as a re-run of the original program. This is the algorithm still taught as the reference implementation of crash recovery, decades on.

The idea sat quietly inside that engineering problem for another twenty years before its epistemic reading surfaced. Event sourcing, and the log-centric architectures associated with systems like Kafka around 2013, made the argument explicit: the log is not a safety net underneath the real system. The log is the real system. State is what you get when you fold over it. This reframing did not change the mathematics. It changed what people thought the log was for.

The turn

Set that reframing next to a question that has nothing to do with databases: what is a model allowed to know, and when is it allowed to know it changed its mind?

A Large Language Model is trained on a corpus frozen at some cutoff. Every document in that corpus contributed to the weights, but the contribution is not addressable. You cannot ask which sentence produced a given belief, or what the model would think if one document in the training set were retracted, because the mapping from source to belief was thrown away the moment training finished. The corpus is a fold with the log discarded. This is not a minor limitation. It means correction requires retraining, and provenance — when anyone bothers to reconstruct it — is done by inference after the fact, not by lookup.

A Large World Model improves on this by being live: it senses a scene continuously, frames arriving in real time, contact and odometry data streaming in while the episode runs. For the duration of that episode, there is a genuine sequence, and belief about the scene really is accountable to it — you can, in principle, ask what changed and why, while it is happening. But the sequence is usually treated as a buffer, consumed and discarded once its content has been folded into the current state estimate. When the episode ends, the accountability ends with it. The world model knows what it currently believes. It does not know, after the fact, what made it believe it.

A Large Universe Model is defined by intake that never stops. That single property forces a question the other two generations can avoid: if the stream never ends, how does belief survive contradiction without either freezing at some arbitrary cutoff or collapsing every time new evidence disagrees with old? Write-ahead logging is the structural answer that already exists, in a different field, for exactly this question. Keep the events. Treat belief as a fold over them. Revision is not retraining and it is not overwrite — it is replay, with the offending event corrected or removed, over a history that still exists to be replayed.

This is why provenance is not a feature you add to continuous intake. It is the condition under which continuous intake is anything other than an unfalsifiable stream of assertions arriving quickly. An always-on system without a log is not more informed than a static one. It is a rumour with good latency.

Why this is a terminal rung, not just a better one

The claim about the intake axis is not "more data is better," which is trivially true and uninteresting. It is that once a system retains the ordered sequence of observations together with their sources, there is no further category of epistemic access to acquire. Any later belief becomes a fold you can recompute. Any retraction becomes a replay with one event struck out. Any disagreement between two parties resolves, in principle, to a locatable point in the log where their histories diverged. There is no fourth position beyond "continuous observation plus the retained history of that observation," because the history of observation already is the meta-level — it is what lets you inspect the process that produced belief, not just the belief. Everything past this point is engineering: how long you retain, how fast you ingest, what it costs per event, how much you trust the recorder. Those things improve indefinitely. The category does not change.

Where this needs narrowing

Three objections earn real space here, and one of them genuinely cuts the claim down to size.

Retention is not free, and at any meaningful scale it cannot be total. Systems compact, snapshot and expire old segments, and a Large Universe Model that must forget to stay affordable ends up with partial provenance — arguably the same position as a language model with the receipts mostly gone. The reply narrows rather than dismisses: there is still a difference in kind between a system that never had a sequence and one that had it and discarded parts of it under a stated, inspectable policy. PostgreSQL deletes WAL segments after archiving; the retention rule is itself a fact you can audit. The honest version of the claim is not total recall. It is that belief carries a pointer to evidence, and where the evidence has expired, the system says so rather than pretending otherwise. Calibrated ignorance is not the same failure as untraceable confidence.

Sequence itself is not always well-defined once observation is distributed across many independent sources. There is no global clock; Lamport's and vector clock formalisms give partial orders, and some events are genuinely concurrent, not merely unobserved in order. An audit trail assembled from thousands of streams may not support one canonical replay. This is correct, and it is the objection that most narrows the thesis. Continuous intake gives you a partial order, not a timeline. What it still gives you, which a frozen corpus does not, is a causal structure — which observations could have influenced which — with genuine concurrency marked as such rather than silently resolved. Systems like Spanner buy something close to total order using atomic clocks and published uncertainty intervals; the interval is disclosed, not concealed. Where order really is undetermined, the correct output is a set of consistent histories with the ambiguity stated, not a false single sequence.

Finally: logging every input does not make inference itself auditable. A trained model's output depends on initialisation, batch order, floating-point nondeterminism, hardware — a deterministic fold over logged events is not what actually produced the belief. This gap is real and should not be argued away. Replay of a logged input gives reproducible inputs, not an interpretable mechanism. What it buys is narrower: counterfactual leverage — retrain or re-fold without a retracted source and measure the difference. That is causal attribution at the level of evidence, which is what retraction and correction actually require in practice. Interpretability of the fold itself is a separate, unsolved problem, and the log does not solve it. It just ensures that when someone eventually does solve it, there is evidence left to interpret.

What this does and does not establish

If the model just kept everything, its beliefs would be true.

Disown that reading explicitly. An unindexed, unsummarised log is inert. Volume does not adjudicate contradiction; it multiplies the number of contradictions on file. Write-ahead logging guarantees nothing about the accuracy of what was observed, the honesty of the recorder, or the soundness of any inference drawn from the fold. What it guarantees is narrower: that a belief can be traced to the evidence that produced it, and revised by replaying that evidence rather than by fiat. Auditability is a precondition for correction. It is not correction itself, and it never claims to be.

Aviation recorders make the boundary concrete. Flight data recorders sample hundreds of parameters continuously, roughly 25 hours of history under the ARINC 717 standard. When Air France 447's recorders were recovered from 3,900 metres in 2011, investigators did not query the aircraft's final state — that state was wreckage, uninformative by itself. They replayed the sequence: pitot icing, stall, sustained nose-up input, in order. The log did not prevent the accident. It is the only reason the accident became explicable rather than merely tragic.

That is the ceiling of the claim. Continuous intake with a preserved, provenanced sequence is the terminal position on the axis of what a system may observe and later account for. It is not a terminal position on truth, on interpretability, or on whether the system is any good at reasoning over what it retains. Those are separate ladders, still being built.