Run It On The File People Actually Have
This found more real defects than every other check combined. A tool passes its own fixtures because the fixtures were written next to the code. Hand it something from the real world and it falls over.
tensorwarden scans model checkpoints for dangerous pickle opcodes. On a real PyTorch file it reported four critical findings, because GLOBAL, REDUCE and BINPERSID are how any tensor is rebuilt. A scanner that rejects 100% of legitimate files does not get fixed, it gets switched off. structuregrade failed every real protein: its clash screen was flagging the peptide bond that holds a backbone together. splatgrade could not open a single real splat file, because every exporter writes binary PLY and the parser was ASCII-only. codecguard could not encode Opus at all: it took the output file extension from the codec name, so it produced a .libopus file that no container can hold.
The fix in each case was the same shape: commit a real artifact as a fixture and make CI run against it. Real AlphaFold models for structuregrade, a real state_dict.pt and a pickled full model for tensorwarden, the same scene in both PLY encodings for splatgrade, real ffmpeg encodes across five codecs for codecguard.
The Benchmark Could See The Answer Key
The second pattern is a method that quietly reads the ground truth it is supposed to predict. The score looks excellent because it is not a prediction at all.
graphpulse scored 0.938 AUC detecting graph anomalies, and the feature it leaned on was derived from the label. Scored fairly it gets 0.66, which is still a real result and a far more useful one. distractrack reported 100% identity tracking while reading the is_target field directly. binauralbench claimed it removed 99.8% of spatial error using a method that was handed the clean target to compare against, so it measured gain recovery from an oracle, not separation. foldcontact and pangraphmap both scored a perfect 100% on quantities that were true by construction.
The tell is a score that is suspiciously round or suspiciously high. I rebuilt each of these as a version that only sees what a real caller would have, published the lower number, and kept the original alongside it so the gap is visible.
A Number Nobody Can Rerun Is A Claim
Several READMEs quoted held-out results from sets that were evaluated once and never committed. guardrail-gate and tablextract both did this. I could not reproduce either number, and neither could anyone else, which makes it a claim rather than a measurement.
I wrote fresh held-out sets in domains the tuning data never touched, froze the code, ran them once, and committed the set along with the result. guardrail-gate's honest numbers came out lower than the ones it had been quoting. That is the point: a held-out number you can rerun is worth more than a better number you cannot.
The Fix Was In A Module Nothing Imported
Twice I found a corrected implementation sitting in a parallel file while the shipped entry point still called the broken one.
safepathshield is a safety filter for control commands. Its continuous-time projection collided in 21 of 40 tuning scenarios and 11 of 20 held-out ones at realistic control rates, and the discrete-time version that holds was in the repo but nothing used it. unlearnaudit reported a leak that was an artifact of a nearest-neighbour self-match, and the corrected scoring was likewise parked in a second module. Both CLIs, APIs and browser UIs now default to the fixed path, with a --legacy flag that reproduces the published numbers exactly.
connectpuct was the same mistake wearing different clothes: the browser game shipped a three-line stand-in weaker than both baselines, while the benchmarked agent lived in Python. Anyone who played it was not playing the thing the benchmark measured.
Sensitivity Without Specificity Is Not A Result
A detector that fires on everything catches every fault and is worthless. Roughly a third of these projects demonstrated only the catching half.
So I measured the other half. spatialniche now proves its null is calibrated: on 40 structureless tissues, 6.2% of z-scores clear the 1.96 threshold, against 5% expected. physicsvideo reports detection and false-alarm rates across 400 randomized clips rather than five hand-built cases. phenopacketlint is measured against nine packets with planted defects and a clean one that must stay silent. strandshift proves a genuinely invariant predictor passes its gate with a range of exactly zero.
Writing those checks surfaced its own trap worth naming. In physicsvideo, dropping frames from the end of a track is an object leaving the scene, not a permanence violation. Labelling it as a fault would have manufactured a detection rate out of nothing.
One Seed Is An Anecdote
methyloadapt reported going from chance to 100% accuracy. Across 90 seeds that turned out to be one lucky draw. Reweighted properly it never does worse than the baseline, which is a defensible claim where the original was not. fedcal's calibration step usually hurt its worst client; the published seed was a favourable draw out of 60. chronopatch's 15% gain came from a series that happened to match its own hardcoded assumptions.
None of these needed a new method. They needed the experiment run more than once, and the honest summary reported instead of the best run.
What It Actually Cost
Every project here now runs its tests, its benchmark and a Docker or UI smoke test in CI, on the Python versions its README claims. Several of those smoke tests immediately caught things nobody would have noticed: a dashboard that crashed on its own page in Docker, an API that returned a server error on the UI it shipped with, a workbench that served its page and then 404ed the engine that page needs.
Plenty of numbers went down. The reranking gain in citebench vanished entirely once I renamed the documents, because it had been reading filenames. The drift detector in churnfm never fires on the drift it was built to catch. ragpoisonbench's provenance defense does literally nothing against an attacker who can get a trusted label, because it never inspects content.
I would rather publish the smaller number with the command that reproduces it. Every figure on this site now has one, and every repository fails its own build if that stops being true.