A Delete Endpoint Proves Nothing
Removing rows from a training table does not establish that a released model forgot them. A useful audit must test residual membership signal and verify that retained behavior did not collapse.
unlearnaudit provides a deterministic k-nearest-neighbor harness with an exact-retraining reference, a membership-inference AUC measurement, and before-versus-after held-out utility.
Measure Privacy And Utility Together
One structured report captures forget and retain sample counts, leakage reduction, test accuracy delta, method, pass criteria, and scope. The CLI, API, workbench, and tests share that audit function.
The Demo
Membership AUC falls from 1.000 before deletion to 0.481 after exact retraining, a leakage reduction of 0.519. Held-out accuracy remains 0.988, so the audit passes with zero measured utility loss.
This small k-NN harness demonstrates the audit contract. It is not proof of deletion for an arbitrary model, optimizer, checkpoint lineage, or threat model.
Update: that 1.000-to-0.481 drop turned out to be a mathematical certainty, not a measured leak. Once corrected, no real membership signal survives before or after retraining — a finding worth reporting honestly rather than papering over. Full writeup below.
Research Basis
The project is informed by OpenUnlearning, a NeurIPS 2025 benchmark that standardizes datasets, evaluation protocols, and privacy-utility assessment for machine unlearning.
Read the OpenUnlearning at NeurIPS 2025.
The Leak Was A Self-Lookup Artifact
The "before" membership score works by finding each forget point's nearest neighbor in a training set that already contains that exact forget point. A point's nearest neighbor in its own training set is always itself, at distance zero — confidence exactly 1.0, guaranteed by construction, for any seed, any noise level, any sample count. Checked directly across 90 seeds (60 tuning, 30 disjoint holdout): that 1.000 never moved once, zero variance. It was never demonstrating a model that memorized its training data; it was demonstrating that a point equals itself.
The fix uses leave-one-out lookup: when a query point is present in the training set, its own copy is excluded before searching for the nearest neighbor, the same trick leave-one-out cross-validation uses to avoid testing a model on itself. The corrected before_auc lands at 0.487 mean across 60 tuning seeds, confirmed at 0.496 on a disjoint 30-seed holdout evaluated once — both essentially chance, nowhere near the published 1.000.
Here's the part I didn't smooth over: after_auc doesn't meaningfully differ from the corrected before_auc on either sweep (0.494 vs 0.487 tuning, 0.511 vs 0.496 holdout). There was no detectable membership leak to begin with in this synthetic 2D k-NN setup, so "exact retraining fixed it" isn't a claim this data actually supports once the tautology is removed — and that's the honest result, not a story rescued by re-tuning. The original module is untouched and the published 1.000/0.481/98.8% numbers still reproduce exactly.