Your RAG Cites Sources Like a Student at 2am
There is a footnote. It looks official. Nobody actually checked whether the passage says what the answer claims. In a term paper that costs you a grade. In an FDA submission it costs a lot more, and "it sounded right" is not a defense anyone accepts.
Here is the trap. Plain keyword search gets fooled every single time by a dead document. A rejected draft that happens to repeat your exact query words will outrank the real, current passage, because it looks like a better match and has no idea it was supposed to stay buried.
citebench is a small, readable RAG pipeline (BM25, dense fusion, reranking) built for one purpose: to measure, in a number, exactly how much reranking rescues you from that failure.
Three Stages, Each One Measurable
It is built in layers so you can see what each one is worth.
• Lexical: a pure-stdlib Okapi BM25 index. The keyword baseline that gets fooled.
• Dense fusion: a hashing dense index blended with BM25 scores, alpha-weighted.
• Reranking: a cross-encoder-style scorer that punishes superseded and draft sources and rewards real phrase overlap over cheap keyword stuffing.
• Validation: a hard confidence floor. If the top citation cannot clear it, the pipeline says "no confident citation" instead of confidently pointing at garbage.
A Benchmark That Fights Back
A benchmark is worthless if the corpus is polite, so I built the traps by hand. A rejected draft that keyword-stuffs the exact query. A superseded manual. The zombie documents that refuse to stay dead in any real archive. Take those out and BM25 looks brilliant and the whole exercise means nothing.
The other rule was zero dependencies and zero API keys, so anyone can clone it and reproduce the number in under a minute without renting an embedding API to fact-check my claim.
The Number
On the adversarial set, keyword retrieval sits at 62% citation precision whether or not you bolt dense fusion on top. Add reranking and it jumps to 88%.
Then I Renamed the Files
I wanted to know how much of that 26-point gain was real. The answer is none of it.
Replace protocol_v2, amendment_v3_draft, and site_manual_v1 with DOC-2210, DOC-3088, and DOC-8004, change nothing else, and precision falls straight back to 62%. The passage text is byte-identical between those two runs. 62% is also exactly the score with reranking switched off entirely, so the reranker was contributing nothing once you took away the filenames.
Because that's what it was reading. The "supersession prior" I'd written was a substring test on the document name against _v1, _v2, draft, old, superseded. And I had named all three distractors in my own corpus to match. I wasn't measuring whether reranking could spot a dead document. I was measuring whether it could read the crib sheet I'd handed it.
Real archives don't label their dead documents helpfully. They use record IDs, content hashes, or the same filename in a different folder. What they do carry, in the text, is the reason the passage is dead: "This definition was superseded on 15 March 2026." "This proposal was rejected by the IRB and never took effect." So the rewrite reads status language out of the passage and keeps the filename only as weak corroboration.
Direction turned out to be the hard part, and it cost me the right answer for a while. The document that retires a requirement and the document being retired share nearly all the same vocabulary. The current amendment says "this section replaces the endpoint definition previously in force," and my first keyword list read "replaces" and "previously in force" and marked the current passage dead. Anchoring the patterns to passive, self-referential forms, "was superseded," "is rescinded," and treating the active forms as evidence of currency instead, is what fixed it.
It Also Couldn't Return Nothing
Q: "Who is the principal investigator?"
A: a stability-testing passage. validated=True, confidence 0.39.
Three of four questions with no answer anywhere in the corpus came back with a validated citation. For a project whose stated premise is that FDA submissions need citations that actually hold up, that's the failure that matters most: a reviewer seeing a passage reference treats the claim as sourced.
The cause was that validation compared the reranker's own score against a floor of 0.2, and that score is a topical similarity, not an answer-support judgment. Ranking asks "which of these is most relevant." Support asks "is the best one good enough to cite at all." Those are different questions, and conflating them is what let a passage about stability testing get returned as evidence about staffing. They're gated separately now, and refusals carry the reason plus every candidate that was rejected and why.
Measured Without the Crib Sheet
On a corpus where every document is an opaque record ID, supersession is stated only in prose, and five questions have no answer at all: 62% → 100%, fabricated citations 5 → 0.
The row I actually care about is the third one: with filename hints switched off entirely, the new version scores identically to the version that's allowed to use them. That's the check that proves the content signals are carrying the result rather than quietly leaning on the same crib the old one was reading. I built those rules against that corpus though, so a separate corpus written afterwards with the code frozen, run once: 50% → 100%, fabrications 3 → 0.
44 tests, and CI now fails the build if the gain ever stops surviving blinding. That last part is the real fix. The original number wasn't a lie, it was measured correctly. It just wasn't measuring what I thought.