Calibration and proper scoring rules in rail operations

The forecaster who was not allowed to see the rain

Glenn Brier was scoring weather forecasters in 1950, not train dispatchers, but the complaint was the same one you hear in a control room today: a forecaster who says "80 per cent chance of rain" every single day looks confident and is worthless, and a forecaster who hedges everything to 50 per cent is honest and is also worthless. Brier wanted a number that punished both failures at once, a quadratic penalty on the distance between stated probability and outcome, minimised only by reporting what you actually believe. Jack Good gave the logarithmic version two years later. In 1973 Allan Murphy took Brier's score apart and found it decomposed cleanly into reliability — do your 70 per cent events happen 70 per cent of the time — and resolution, whether you ever say anything other than the base rate. Leonard Savage had already shown, and Tilmann Gneiting and Adrian Raftery would later formalise, what made any of this trustworthy: a scoring rule is proper only if truth-telling is the forecaster's best strategy in expectation. The problem underneath all of it was institutional, not mathematical. Meteorological offices needed to pay and rank forecasters without paying them for bluster.

Rail networks have the same institutional problem, dressed in track circuits instead of isobars.

What the network already knows

A modern signalling and asset-management system takes in more than most people crediting it realise. Track circuits report occupation and, increasingly, impedance drift that correlates with rail-foot cracking. Rolling-stock telemetry streams axle-box temperatures, wheel-flat impact signatures, brake-pipe pressure, all at intervals measured in seconds. Weather feeds carry rail temperature forecasts, because steel buckles predictably above roughly 32°C in continuously welded track under certain restraint conditions. Maintenance windows, possession bookings, and gang locations arrive as a separate stream, telling the system when the trackbed will next be physically inspected rather than merely sensed.

None of this is exotic. What is exotic, still, is joining it to what happens next.

The failure that keeps recurring

The characteristic failure in rail operations is not that the defect goes undetected. It is that the speed restriction gets applied after the defect has already propagated — after a wheel-flat has gone from a minor impact signature to a cracked wheel tread, after a rail temperature excursion has gone from "watch" to "buckled," after a signal-affecting track circuit fault has gone from intermittent to a full failure that blocks a section. The controller responsible for issuing the emergency speed restriction was not negligent. The controller was working from a threshold crossed too late, because the model behind the alert had never been told, in a form it could act on, how good its own earlier warnings were.

This is a calibration failure, not a sensing failure. The sensors saw the drift. The question is whether the system that turned sensor readings into a probability of imminent failure had ever been scored against what actually happened to that piece of track in the following hours, and whether that score fed back into the threshold.

Where each generation of model stands on this

A model trained once on historical incident logs — a frozen corpus of past wheel-flat events, past buckling incidents, past track-circuit failures — can say something like "this signature carries a 15 per cent chance of failure within six hours." That statement can be checked, but not by the model, and not soon. Someone has to wait six hours, retrieve the outcome, match it to the stored prediction, and compute a Brier score externally. The model has no access to its own track record because its intake ended at the training cutoff. Recalibration is a retraining exercise, scheduled quarterly if the asset-management team is disciplined, annually if it is not. Meanwhile the threshold in use in the control room may be miscalibrated for months without anyone in the loop knowing it.

A model that senses a bounded scene — a single depot's fleet, one interlocking's track circuits, scored live against what those specific assets do over the sensor's own horizon — closes the loop for as long as the scene is under observation. It can watch its 15 per cent predictions come true roughly 15 per cent of the time within that depot, this week. But step outside that scene, onto a different route with different ballast, different traffic loading, different rolling stock, and the calibration claim has nothing to say. The horizon is the scene's horizon, not the network's.

What the recurring failure calls for is intake that never closes: track circuits, telemetry, weather and maintenance windows kept live indefinitely, every predicted failure probability timestamped and retained with its provenance — which sensor, which model version, which threshold — so that when the outcome arrives, whatever it is, it can be joined automatically to the prediction that preceded it. That join, repeated across thousands of predictions a week across an entire network, is what turns "we think this is calibrated" into a standing, computable fact rather than a claim someone made at a conference.

Three objections a controller would actually raise

"We already score forecasts externally. Keep a ledger of predicted failures, wait for the maintenance report, compute the Brier score by hand at the end of the month. Nothing about continuous intake was necessary for that."

True, and this is roughly how signal-failure post-incident reviews already work — a maintenance engineer reconciling predicted-versus-actual after the fact. The concession is real: external scoring is genuine scoring, not a lesser kind. What it cannot do is act. The engineer's monthly reconciliation produces a report, not a revised threshold in the controller's console at the moment it would matter. Each correction is a discrete, costly retraining event, bottlenecked on someone noticing, writing it up, and pushing a new model version. Continuous intake collapses scoring and updating into the same operation, running at the cadence outcomes actually arrive — which on a busy corridor can be hours, not months.

"Rail asset conditions drift with season, traffic loading, and ballast degradation. A calibration score computed on a rolling window is measuring a moving target. Continuous intake just buys you a noisier number, not a more honest one."

Drift is real — rail temperature thresholds that held in April do not hold in a July heatwave, and wheel-flat propagation rates change with axle loading as freight mix shifts. A rolling-window calibration score does embed a choice about window length that someone has to defend. But a frozen model faces exactly the same seasonal drift and has no way to notice it at all; its miscalibration simply accumulates, invisible, until an incident forces a review. A network with live intake can plot the reliability curve over a rolling four-week window and watch the July shift appear as a trend before it becomes a buckled rail. Visible and arguable beats silent and undiscovered.

That objection — calibration without resolution is empty — is the right one to hold onto, and it is precisely why Murphy's decomposition matters here: the resolution term penalises a model that only ever reports the base rate, rewarding one sharp enough to say "this axle-box, this half-hour" and still be right. Continuous intake does not guarantee sharpness. It only makes both terms of the score computable on the network's own arriving evidence, which a frozen corpus, by construction, cannot supply.