Particle filters and sequential Monte Carlo in security operations

The objection that should win

Here is the strongest version of the case against this page, stated the way a sceptical detection engineer would put it.

A particle filter needs a state space you can enumerate and a likelihood you can write down. A security estate has neither. The hidden state is not a ship's position with six degrees of freedom; it is "is this identity compromised, and if so, by whom, doing what, for how long, across which of forty thousand endpoints." The dimensionality is not high, it is unbounded — new hosts, new cloud roles, new SaaS integrations appear weekly, each adding state. Threat intelligence arrives as prose, not as a Gaussian around a measurement. You cannot resample your way out of not knowing what you're modelling. Citing a 1993 tracking algorithm as proof that continuous security monitoring is a solved inference problem is exactly the kind of borrowed mathematical respectability that should make a reader suspicious.

That objection deserves to be taken at full strength before anything is conceded back. It is largely right about the difficulty. It is wrong about what follows from the difficulty.

What a particle filter actually is, and why the SOC needs one

A particle filter tracks a hidden state by keeping a cloud of weighted guesses instead of a single estimate. Each particle is a full hypothesis: this identity is compromised via a stolen session token, that endpoint has a living-off-the-land beacon dormant since Tuesday, this service account's anomalous OAuth grant is legitimate automation. At every timestep, each particle is pushed forward by a model of how intrusions behave — lateral movement follows trust boundaries, persistence mechanisms survive reboots, credential theft precedes privilege escalation by hours, not weeks — and then reweighted against whatever evidence just arrived. Particles that predicted badly lose weight. At resampling, they are discarded; particles that predicted well are duplicated. The distribution itself is the belief. There is no single number for "compromised: yes or no," only a population of surviving hypotheses and their relative credibility.

This is the exact shape of the security operations centre's actual problem, and it is not the shape a Large Language Model or a Large World Model can hold. The Large Language Model performs one batch update against a threat-intelligence corpus frozen at some cutoff and never reweights again; ask it about a technique published last month and it has nothing, because nothing new has arrived to condition on. The Large World Model can run a genuine filter, predict, weight, resample, but only across a bounded episode: an incident response engagement, a single investigation window, a scene that closes when the case is closed. Security operations has no closing scene. EDR telemetry, threat intelligence feeds, identity events and configuration drift do not stop when the analyst goes home. The characteristic failure of the field is precisely the gap a bounded filter cannot cover: an intrusion dwells for the interval between one hunt and the next, because belief about the estate was allowed to go stale between observations instead of being carried forward and continuously reweighted.

Conceding the degeneracy problem

The sceptic's dimensionality point survives contact. Bengtsson, Bickel and Li showed formally that naive particle filters collapse in high dimensions: the number of particles needed grows exponentially with the effective dimension, weights concentrate on a single sample, and resampling then just duplicates that one lucky guess into a crowd of clones with no diversity left to represent uncertainty. A security estate with forty thousand endpoints, thousands of identities and a constantly mutating configuration graph is not a low-dimensional tracking problem. Treating the whole estate as one monolithic state vector and running a textbook bootstrap filter over it would degenerate within a handful of timesteps, exactly as the theorem predicts. Citing 1993 tracking algorithms as if that settles the scale question would be a bad argument, and it should be rejected when made that way.

What survives is the decomposition, not the monolith. Nobody who filters high-dimensional systems in practice filters them as one block. FastSLAM factorises a map into thousands of conditionally independent landmark filters because landmarks barely interact once the robot's own pose is known. A SOC's estate has the same conditional structure. An identity's compromise state is only weakly coupled to a printer's patch level three subnets away; a config-drift event on an isolated dev server barely moves the posterior on a production database credential. Rao-Blackwellised filtering marginalises the tractable pieces analytically and leaves particles only for the genuinely nonlinear, multimodal part of the problem — in a SOC, that residual is usually something like "which of these three plausible attack narratives explains this cluster of anomalous authentications." The honest claim is not that a single filter tracks the entire estate. It is that the estate decomposes into many coupled, mostly local filters — per identity, per asset class, per trust boundary — each small enough that the degeneracy theorem does not bite, wired together where the coupling is real: a domain controller compromise, for instance, correlates simultaneously across every identity filter downstream of it, and that correlation has to be modelled explicitly rather than assumed away.

Conceding the likelihood problem, mostly

The second serious objection is the likelihood one, and it is the sharper of the two for this domain specifically. A particle filter weights hypotheses by how well they predict the next observation, which requires a stated p(observation | state). For a radar altimeter that number comes from a sensor datasheet. For a fresh threat-intelligence report about a novel technique from a vendor with no track record in your sector, there is no datasheet. Prose is not a likelihood function. A SOC analyst reading an intel bulletin cannot mechanically convert "actor X is reportedly targeting sector Y" into a weight on forty particles representing possible intrusion states.

This is real, and it does not fully dissolve. But it is not all-or-nothing either, because reliability itself is estimable, and estimable things can be filtered. A given EDR agent's false-positive rate on a specific detection rule, a given threat-intel provider's historical accuracy against later-confirmed incidents, an identity provider's log completeness during outages — these all have track records. Treat source reliability as part of the state, updated the same way the underlying compromise hypotheses are updated: a feed that has cried wolf on ransomware precursors nine times out of ten has its future weight discounted automatically, the same way an adaptive filter learns to discount a noisy sensor's readings without being told in advance how noisy it is. What remains genuinely irreducible is the brand-new source with zero history — the first report of a zero-day from an unfamiliar research team. That case gets a wide, weakly informative prior and stays wide until corroborated. That is not a failure of the framework; it is the framework correctly refusing false confidence, which is more honest than a dashboard that reports a single risk score to two decimal places regardless of how little is actually known.

The narrower claim that holds

None of this establishes that continuous security monitoring across every stream is a solved engineering problem. Access to identity events across every SaaS tenant, jurisdiction over cross-border telemetry, and institutional trust in an always-on system that never stops reweighting its beliefs about employees — these are governance questions, and no resampling step touches them. A SOC analyst who objects that "the hard part is getting the data feeds and the sign-off, not the maths" is objecting to something true, and this page does not answer it.

What the lineage argument actually needs is narrower and holds up: the inferential form for revising belief under a permanent stream of heterogeneous, unreliable evidence is not speculative. It has forty years of working deployments, in domains that share the security operations centre's structure exactly — multimodal hypotheses, evidence gaps, sources of uneven quality, decisions that cannot wait for certainty. What separates a Large World Model's incident-scoped filter from a genuinely universe-scale one is not a missing algorithm. It is that the streams never stop, the sources vary in trust, and the filter has to carry that trust forward as part of what it believes. Once that is granted, the argument against the third position has to be made on access, consent and governance grounds. It cannot be made on the grounds that nobody knows how to update a belief when new evidence arrives while old evidence is still being weighed. That part was settled before most current SOC tooling existed.