The Coworker Who Asks "Wait, Which Jaguar?"
You know that coworker who answers your Slack question confidently, at length, about the wrong thing entirely? Most RAG agents are that coworker. Ask "what's the price of apple?" and you'll get a confident answer about either the fruit or the company, decided by a coin flip you didn't know was happening.
clarifyrag asks first when it's actually unsure. It gates on retrieval disagreement: if the evidence splits across two comparable senses, it asks. If one sense clearly dominates, it just answers. No committee meeting required, no asking about things that are already obvious.
Technical Architecture
Zero dependencies, an interpretable TF-IDF retriever, and a heuristic gate you can print the reasoning for:
• Ambiguity score: the mass ratio of the runner-up sense to the top sense, where 0 means clear and 1 means a coin flip.
• Lexical backstop: a known polyseme (apple, jaguar, python, mercury) used with no disambiguating term anywhere nearby raises the ambiguity signal even when the retrieval scores alone look borderline.
• Multi-turn resolution: ask, take the user's hint, then answer, tracked through a Resolution object recording how many turns it took and whether the final answer was correct.
• Bring your own retriever: the TF-IDF search swaps out for a real vector DB behind the same interface, with the ask/answer gating logic unchanged.
Challenges & Solutions
The interesting design tension was picking a utility function that actually captures the tradeoff, since accuracy alone rewards an agent that asks about everything, and speed alone rewards one that never asks. Scoring each policy as accuracy minus a per-question penalty made "asks selectively" and "asks about everything" genuinely comparable instead of an apples-to-oranges argument about which metric matters more.
I also made sure the gate's decision is fully inspectable, no black box confidence score. Every ask or answer comes back with the actual sense-mass ratio and a plain-language reason, which matters more for debugging a real deployment than a marginally higher F1 score would.
Impact & Results
On a labeled benchmark of ambiguous and well-specified queries, always-answering scores 77% accuracy at zero questions asked. Always-asking scores 100% accuracy but nags on every clear query. Clarify-aware gating hits the same 100% accuracy while asking 38% fewer questions than always-asking, the best utility of any policy tested.
7 tests pass in CI across Python 3.9 through 3.13, plus the example script and full benchmark run on every push.