A Delete Endpoint Proves Nothing
Removing rows from a training table doesn't establish that a released model forgot them. A useful audit has to test for residual membership signal and verify that retained behavior didn't collapse.
unlearnaudit is a deterministic k-nearest-neighbor harness with an exact-retraining reference, a membership-inference measurement, and before-and-after held-out utility.
Measure Privacy and Utility Together
One structured report carries forget and retain counts, leakage reduction, test-accuracy delta, method, pass criteria, and scope. The CLI, the API, the workbench, and the tests all share that audit function.
The Demo
Residual membership AUC after exact retraining is 0.481, essentially chance, and held-out accuracy holds at 0.988 with zero utility loss, so the audit passes. It's a small harness demonstrating the audit contract, not proof of deletion for an arbitrary model, optimizer, checkpoint lineage, or threat model.
The Leak I Reported Was a Self-Lookup
This page used to lead with membership AUC falling from 1.000 to 0.481, a leak found and fixed. That 1.000 was arithmetic, not a measurement. Membership confidence was exp(-distance to nearest training point), and the before-score asked each forget point about a training set that still contained it. Its nearest neighbor is itself, at distance zero, so the confidence is exp(0) = 1.0, for any seed and any data. Across 90 seeds it was exactly 1.000 every time, with zero variance.
With leave-one-out lookup, which excludes a point's own copy before finding its nearest neighbor, the before-score lands at 0.487 across 60 tuning seeds and 0.496 on a disjoint 30-seed holdout, evaluated once. Both are chance. The after-score doesn't meaningfully differ, 0.494 and 0.511. So the honest reading is that on this harness there was never a detectable membership signal to remove, and "retraining fixed it" isn't a claim the data supports. I'm reporting that rather than re-tuning the setup until a before-and-after story appears.
The corrected scoring is now what the tool actually ships: the CLI, the API, and the workbench default to it, and --legacy reproduces the published numbers. The verdict never rested on the artifact, which is the one piece of the original design I'd defend. A pass requires residual AUC at or below 0.6 with utility intact, not a large delta. 14 tests pass across Python 3.10 through 3.13, and CI fails the build if the shipped audit ever prints that 1.000 again.
Research Basis
Informed by OpenUnlearning, a NeurIPS 2025 benchmark that standardizes datasets, evaluation protocols, and privacy-utility assessment for machine unlearning. Read the OpenUnlearning paper.
Try It
No dependencies:
git clone https://github.com/ahmeddoghri/unlearnaudit && cd unlearnaudit
pip install -e .
unlearnaudit demo # the honest report: before 0.468, after 0.481, passes
unlearnaudit demo --legacy # the published 1.000 → 0.481
python -m unlearnaudit.eval_v2 # 90 seeds: the 1.000 never moves, the corrected score is chance