Checking whether the prediction results are real: two leakage claims and a label-permutation control

2026-08-17 · EEG seizure prediction, preictal window

Follow-on to the first trustworthy prediction-mode results (mean roc_auc 0.882, hit rate 6/7). Strong results on a first real pass at a task existing models handle poorly invited a second round of scrutiny rather than acceptance: an external critique raised two concrete leakage-risk claims about the pipeline, checked directly against the real code and real data; separately, a label-permutation null control was added to test whether the model is learning anything at all, independent of either leakage claim.

Two leakage claims, checked against real code and real data

Claim 1: does fold-boundary exclusion actually remove a held-out seizure's full preictal lead-up from every other fold's training set, not just its ictal period? Claim 2: is the SPH warn-zone boundary airtight, such that no window straddling the preictal/warn-zone edge can slip into the positive class? Both were checked by independent recomputation against the built dataset's own per-window metadata and the real fold-construction code, not by re-reading source and trusting it.

The first attempt at Claim 1 was itself buggy: an initial version compared window start/end times against seizure onset across the entire dataset regardless of recording. That produced thousands of apparent leaks per fold. Root cause: window timestamps are relative to their own recording's start, not a shared clock, so two unrelated recordings both having a window at recording-relative start=2000s means nothing. Fixed by restricting the comparison to windows from the same recording as the held-out seizure, which is what the check was meant to test.

fold	seizure	result
0	1_03_0	PASS
1	1_04_0	PASS
2	1_15_0	PASS
3	1_16_0	PASS
4	1_18_0	PASS
5	1_21_0	PASS
6	1_26_0	PASS

Claim 1 confirmed true, and the mechanism is stronger than the minimum required: the test mask is computed at whole-recording granularity, not by targeting the preictal range specifically, so the entire held-out recording — preictal, ictal, and postictal windows alike — is excluded from every other fold's training set. Claim 2 also confirmed true: 0 of 654 positive windows overlap their seizure's SPH warn zone across all 7 seizures. Exclusion is computed independently of the positive-label condition and given strict precedence in the final label assignment, so a window straddling the preictal/warn-zone boundary is always dropped, never labeled positive.

Neither leakage mechanism raised by the critique is present in the current code. This rules out two specific, plausible failure modes; it doesn't by itself validate the results overall — that's what the null control below is for.

Label-permutation null control

A new --shuffle-labels flag shuffles the label array globally right after dataset construction — class balance preserved exactly, positive/negative assignment turned into pure noise — then runs the identical downstream pipeline: same fold construction, same per-fold train-only subsampling, same classifier, same everything except which windows are labeled positive. Results write to a separate results/prediction_shuffled_control/ directory so a null-control run can never be mistaken for a real one. Verified at smoke scale first (roc_auc collapsed to 0.531, at chance) before running for real, matched exactly to the real run's configuration.

seizure	real auc_pr	null auc_pr	ratio
03	0.520	0.044	11.8×
04	0.295	0.042	7.0×
15	0.677	0.046	14.7×
16	0.956	0.052	18.4×
18	0.187	0.037	5.1×
21	0.003	0.043	n/a — see below
26	0.124	0.038	3.3×

Mean roc_auc: real 0.882 vs. null 0.491 (textbook chance). Mean average_precision: real 0.395 vs. null 0.043 (~9×).

Seizure 21's row isn't a real counter-example. n_preictal=3 prints identically in both runs because that count is read from metadata untouched by the shuffle, but the auc_pr itself is computed against however many shuffled-positive windows actually landed in that fold's test set, which is 77, not 3 — a global permutation of 654 positives across 15,912 windows puts roughly 77 into any ~1,878-window test slice by base rate. So real (n=3, high-variance) auc_pr=0.003 and null (n=77, stable) auc_pr=0.043 aren't a fair comparison in either direction; both numbers are legitimate, they're just answering different-precision questions. Every other fold's n_preictal (89–113) is large enough that this issue doesn't apply.

Six of seven folds show a clean, consistent real-vs-null gap (3×–18×), and the aggregate roc_auc sits almost exactly on the 0.5 chance line for the null while the real run holds at 0.882 — solid evidence the real run's signal isn't a pipeline artifact, on top of, not instead of, the two leakage mechanisms ruled out above.

Open items

Decision threshold is still untuned. Everything so far remains single-subject, patient-specific leave-one-seizure-out — the easier end of seizure prediction; the null control rules out "no real signal at all," not "this generalizes beyond chb01." A stricter complementary check — a circular time-shift control, where the signal stays the same but seizure-onset alignment is randomized instead of the labels — was discussed but not built; it would rule out a narrower failure mode (the model keying on some arbitrarily-timed window shape rather than genuine seizure-onset proximity) that label-shuffling alone doesn't fully address.