A full 6-fold real run — and a real recall cost hiding in the k-of-n win

2026-08-23 · EEG seizure prediction, alarm smoothing

Follow-on to the same day's bf16 speedup and OOM fix. With both fixes in place, the first-ever completed run of this exact configuration — full 23-channel data, uncapped test sets, both bf16 flags on, k-of-n smoothing applied — finished in about 34 minutes with no crash. The headline number looks great until the smoothing step is examined closely.

Per-fold results

Real leave-one-seizure-out cross-validation across all six of chb01's seizures, full-density test sets (650–750 windows per fold, never subsampled).

seizure	preictal	hit (raw→smoothed)	FAR/hr (raw→smoothed)	precision	recall	f1	AUC-PR
03	30	✓ → ✗	8.33 → 2.17	0.038	0.067	0.049	0.108
04	30	✓ → ✓	22.26 → 9.29	0.190	0.900	0.314	0.458
15	30	✓ → ✓	6.45 → 1.22	0.413	0.867	0.559	0.536
16	23	✓ → ✓	1.67 → 0.00	0.667	0.870	0.755	0.803
18	30	✓ → ✗	1.00 → 0.00	0.500	0.200	0.286	0.447
26	30	✓ → ✗	3.40 → 0.00	0.056	0.033	0.042	0.123

Mean across folds: accuracy 0.921, precision 0.311, recall 0.489, f1 0.334, average_precision 0.413, roc_auc 0.916. Event-level hit rate: 6/6 (100%) raw, dropping to 3/6 (50%) after smoothing. False-alarm rate: 7.19/hr raw, down to 2.11/hr smoothed — a ~70% reduction.

The tradeoff those two headline numbers hide: k-of-n smoothing erased three real detections (seizures 03, 18, 26) along with the false positives it was meant to suppress. Those seizures' preictal windows genuinely were flagged by the raw classifier — the positive predictions just never sustained the required 8-of-10 consecutive-window density to register as a smoothed alarm. Cutting false alarms by 70% cost half the event-level hit rate. That's a real recall cost on this subject, not a free win.

Why this didn't show up earlier

Two smaller diagnostic checks run earlier the same day — a quick sanity-scale run and a comparison against the Truong baseline itself — both showed 6/6 hit rate at both raw and smoothed thresholds, no apparent cost. Neither of those used this run's real, full-density test sets; the tradeoff only became visible at real scale. The smoothing window (k=8 of n=10) was inherited as-is from the published baseline's own paper defaults, never independently tuned for this pipeline's own window and step size — whether that's actually the right smoothing window here, or needs its own tuning pass, is now a genuinely open, undecided question rather than something to assume is fine because it came from a credible source.

Open items

The k-of-n tuning question above needs a deliberate decision, not just a bigger run. This is subject 1 (chb01) only — no claim here about how any of this generalizes to other subjects. Neither this run's code (the bf16 and OOM fixes from the companion session) nor its results are committed yet.