Citing Sources Like a College Essay at 2am
Everyone cites sources like a college essay written at 2am: technically there's a footnote, nobody checked if it actually says what you claimed. In FDA regulatory submissions that gets people hurt, not just a bad grade. "The answer sounds right" isn't good enough. The cited passage has to actually be the current, correct one.
Plain lexical retrieval gets fooled constantly. A superseded draft that repeats the query's exact keywords will outrank the real, current passage every time, because it looks like a better keyword match and has no idea it's supposed to be dead. citebench is a small, inspectable RAG pipeline (BM25 plus dense fusion plus reranking) built to measure exactly what reranking buys you against that specific failure mode.
Technical Architecture
The pipeline runs three stages so the effect of each is separately measurable:
• Lexical retrieval: a pure-stdlib Okapi BM25 index over the passage set.
• Dense fusion: a hashing-based dense index fused with the BM25 scores using min-max, alpha-weighted combination.
• Reranking: a cross-encoder-style joint scorer that penalizes superseded or draft sources and rewards genuine phrase overlap over raw keyword repetition.
• Citation validation: a hard confidence floor; if the top citation doesn't clear it, the pipeline reports "no confident citation" instead of guessing.
Challenges & Solutions
Building a fair benchmark meant deliberately engineering the adversarial cases instead of hoping the corpus would surface them naturally: a rejected draft amendment that keyword-stuffs the exact terms of the query, a superseded site-manual version, the zombie documents that just will not stay dead in a real document set. Without those specific traps, BM25-only retrieval looks fine on paper and the whole point of the benchmark disappears.
The other constraint was keeping it zero-dependency and zero-API-key so anyone could clone the repo and reproduce the number in under a minute, no embedding API bill required to check whether the claim is honest.
Impact & Results
On the bundled adversarial benchmark, BM25-only retrieval holds citation precision at 62% whether or not dense fusion is added. Adding reranking on top lifts it to 88%, an increase entirely attributable to the reranking step penalizing superseded and keyword-stuffed sources.
7 tests pass in CI across Python 3.9 through 3.13, along with the example script and the benchmark itself run on every push.