A compact cache, a k-sweep, and a dead encoder removed

2026-08-24 · performance, ablation, GPU

One continuous Windows/CUDA session with three chapters: a caching bug that made a 100%-hit cache slower than no cache at all, a systematic sweep of how many live coherence edges the model actually needs, and the discovery that an entire encoder module had been running, uselessly, on every batch for over a week.

A cache that hurt more than it helped

The live-clique channel-subset design (previous entry) scatters a handful of real coherence edges into an otherwise-zero tensor sized for the full 253-edge mesh. The disk cache had been storing that full-size tensor as-is — meaning a cache read for a 6-edge live clique was still loading a file that was 97% zeros. On this GPU, reading that file took about 26 milliseconds; actually recomputing the same 6 live edges from scratch took about 2.3 milliseconds. The cache had become slower than not having one. Fixed by storing only the nonzero edge slots on disk and scattering them back to full size on load — cache files shrank from 15.5MB to 0.37MB, and disk usage for the full dataset dropped from roughly 12GB to 0.28GB. A second fix skipped the wavelet transform entirely on a batch where every trial was a complete cache hit, since it had been computed and then silently discarded on every cache hit until now.

config	epoch time	what was happening
original (100% cache hits)	27.87s	loading 15.5MB files, mostly zeros; transform still ran
after compact rewrite	11.06s	fat files treated as misses, cheap recompute, compact files written
after compact + skip-transform	8.61s	100% compact hits, transform skipped entirely

The right policy turned out to be device-dependent, not universal. On Apple Silicon, the same fat 15MB files are cheap to read after a couple of epochs thanks to unified-memory page caching, so treating them as a miss there would force an unnecessary recompute. On this CUDA machine, recompute is cheap enough that a fat file is genuinely better treated as a miss. Both behaviors are now handled by one explicit device-dependent switch rather than a single hardcoded policy. Following the same logic to its conclusion, the disk cache is now off by default on CUDA entirely for the full, non-subsetted mesh too — consistent with an earlier finding on a rented GPU pod that removing the cache altogether cut training time by roughly a third there as well.

How many live edges does the model actually need?

With the cache question settled, a systematic sweep across k = 4, 8, 12, 16, and 20 live channels (6 up to 190 live edges, out of a possible 253), on the real uncapped dataset across all six seizures:

k	live edges	~s/epoch	AP	AUC	k-of-n hits
4	6	3.9	0.167	0.809	5/6
8	28	4.4	0.186	0.854	4/6
12	66	6.2	0.457	0.932	6/6
16	120	9.2	0.534	0.950	5/6
20	190	12.1	0.567	0.953	5/6

Ranking quality climbs steadily with more live edges — k=20 edged out the documented full-23-channel run on both AP and AUC, and came close to matching the Truong baseline's own AP on this data. But event-level hit rate after smoothing doesn't move in the same direction: k=12 is the only setting that caught all six seizures after smoothing, while both k=16 and k=20 missed one that k=12 caught. More live edges gave better ranking, not simply a better detector. Larger k also stopped earlier via early stopping, which is why the wall-clock gap between k=4 and k=20 was only about 2× despite the per-epoch cost being roughly 3×.

Not a clean k-vs-full-mesh comparison as run: this sweep used a held-out validation split with early stopping, while the documented full-23-channel baseline trained on 100% of each fold for all 20 epochs with no early stopping. The gap between k=20 and full mesh above is partly a training protocol difference, not purely an edge-count effect — flagged directly as needing a same-protocol full-mesh rerun before treating it as settled.

An encoder that had been running for a week and doing nothing

Digging into where epoch time was actually going surfaced something unrelated to caching or edge count entirely: a per-channel raw-signal encoder module had been constructed and run on every single training and prediction batch since a configuration change over a week earlier — two dilated convolutions over the raw 23-channel signal, computed in full — and then its output was discarded and replaced with zeros before ever reaching the classifier. Because that replacement used a detaching zero-fill, no gradient flowed through it either. The module was pure wasted compute, silently, the whole time. Removed entirely rather than left in place: the classifier's input layer shrank accordingly, and every model trained before this fix is now a different shape and can't be reloaded — a clean break rather than a compatibility shim.

Removing it did not meaningfully speed anything up, which was itself informative: the module's own weights were tiny, and what disappeared was peak memory from its intermediate activations, not a persistent cost. Steady-state epoch time at k=4 and k=20 landed within noise of the pre-removal numbers. The real remaining cost, confirmed directly, is the wavelet transform running on all 23 channels every batch regardless of how few edges are live — edge count changes the coherence computation's cost, not the transform's.

A matched fp32-vs-bf16 comparison run partway through (before the removal) reinforced an earlier finding: thresholded metrics matched exactly between precisions, continuous ranking scores differed by about a hundredth of a point, and epoch time roughly tripled without bf16 — the speedup flags are a clear, safe win with no accuracy cost at real scale, even though they don't show up as a difference at smoke scale, where the GPU never gets full enough for the precision choice to matter.

Open items

A same-protocol full-mesh rerun (validation split, early stopping, encoder-free) is needed before the k-sweep's win over "23 channels" can be trusted as a real edge-count effect rather than a training-protocol artifact. An in-memory compact cache for the live-edge tensors is flagged as the next real lever — disk already lost to recompute on this hardware, and the full-precision tensors at higher k don't comfortably fit in this card's 8GB of VRAM. The k-sweep's own AP table was run with the now-removed dead encoder still in the graph and would need a rerun to be the encoder-free numbers.