OOM fix, a streaming classifier, and the first trustworthy prediction-mode results

2026-08-17 · EEG seizure prediction, preictal window

First real (non-smoke) run of prediction mode — predicting the preictal window, not just classifying active seizure state — on chb01, leave-one-seizure-out, 7 folds.

OOM crash

--label-mode prediction --subjects 1 --epochs 10, real step_size=8, crashed (exit 137). Diagnosed via vm_stat/sysctl vm.swapusage (12.6GB swap in use on a 16GB+15GB-swap machine) and by measuring cache-entry sizes directly: dense_edge_cache ≈2.04MB/window, cwt_window_cache ≈1.51MB/window — ≈3.55MB/window combined. Feature preparation runs once for the whole training set before the epoch loop starts. At prediction mode's scale (≈14,700 training windows/fold, since a seizure-containing recording can't supply its own negatives, so the negative pool is the entire rest of the subject), that's ≈52GB materialized before training begins. Not a leak — an architectural mismatch between "materialize everything up front" (fine at detection's ≈700-window scale) and prediction's real dataset size.

Two fixes

Stratified negative-window subsampling. Negative windows randomly dropped to 5× the positive count, stratified per (subject, run) so every recording keeps a proportional share. All positive (preictal) windows always kept.

Streaming classifier. A new class, subclassing the existing detection classifier and overriding only fit(): computes features per batch instead of once for the whole training set — O(batch size) memory instead of O(dataset size). Detection's classifier and code path left untouched.

Bit-exact reproduction of the original class's training trajectory turned out not to be achievable (a shuffled data loader draws an extra seed from the shared generator before sampling, shifting subsequent output). Verified instead on what actually matters: every training index visited exactly once per epoch, per-batch feature output numerically identical to whole-set feature output for the same windows (0.0 absolute difference), and end-to-end fit/predict sanity. Both fixes verified together at smoke scale before the real rerun.

First real run — later found confounded

~1h32m, exit 0, all 7 folds. Mean roc_auc 0.868, average_precision 0.552, hit rate 6/7. The numbers looked strong enough, for a first pass at a task existing models don't do well on, to be worth re-checking rather than accepting. Re-reading the fold-construction code end to end surfaced a real bug: negative-window subsampling was being applied once, globally, before the fold split, so every fold's test interictal windows were also thinned to the same ≈1/5 ratio as train. false_alarms_per_hour and the window-level metrics were being computed against a random ≈21% slice of each recording, not continuous monitoring. No train/test leakage — fold construction itself checked out clean — but a real reliability problem with the sample sizes behind the metrics that matter most.

Fix: subsample train only, per fold

Subsampling moved out of the shared dataset-building step and into the per-fold loop, applied to each fold's training slice only, after the train/test split. Test windows are now never touched by subsampling, at any ratio. Verified at two subsampling ratios: at the default ratio, smoke scale was already too small to trigger subsampling (expected); at a ratio forced to actually engage, the subsampling fired once per fold with different counts (confirming it operates on each fold's train-only pool, not a shared global one), while every fold's test count matched the unsubsampled baseline exactly.

Second real run — the numbers that stand

Same command, ~1h53m (longer than the first run — the previously-dropped ≈80% of each fold's interictal windows needed fresh feature-cache entries). Exit 0, all 7 folds.

Mean: accuracy 0.920, precision 0.380, recall 0.512, f1 0.413, average_precision 0.395, roc_auc 0.882, false_alarms_per_hour 59.0, hit rate 6/7 (85.7%).

Against the confounded run: roc_auc held up (0.868 → 0.882) — the ranking signal was real, not an artifact of the eval bug. average_precision dropped substantially (0.552 → 0.395) once scored against the true ≈20× larger interictal population per fold. FAR/hr moved in both directions per fold after the fix (down for one seizure, up for another), consistent with the earlier subsampling being unbiased noise rather than a directional inflation.

On why a false-alarm rate as high as 97.6/hour doesn't by itself mean there's no signal: false-alarm rate, precision, and recall are all read off one fixed, untuned decision threshold, while roc_auc and average_precision are threshold-independent rank statistics. For seizure 04's actual class balance (≈4.8% positive), a zero-skill model would score auc_pr≈0.048; the observed 0.295 is ≈6× that floor — real signal, poorly exploited by an uncalibrated threshold, not absent.

Open items

Decision threshold is untuned — every false-alarm number above is read off the default class-weight-skewed cutoff; sweeping it is a plausible cheap lever, not yet tried. Everything so far is single-subject, patient-specific leave-one-seizure-out — the easier end of seizure prediction; cross-patient generalization is untested. Prediction-mode hyperparameters are still a straight copy of detection's tuned numbers, with no prediction-specific tuning done.

Update: a chance-baseline significance test and two specific leakage-risk claims raised by an external critique were checked directly against the real code and real data — see Checking whether the prediction results are real.