Function Calling Works Great at Five Tools. It Falls Apart at Fifty
With five tools an agent picks correctly almost every time. With fifty it starts firing the wrong tool, inventing arguments, and doing it all with the exact same serene confidence it uses when it is right. A wrong tool call is not a harmless mistake either. It sends the email. It writes the file. It runs the query.
toolrouter is the missing piece: a selector that knows when it does not know. It scores confidence as the margin between the best tool and the runner-up, and abstains when that margin is thin instead of guessing.
The Margin, Not the Score
This is the one idea that matters. Confidence is not how well the top tool scored. It is how much it beat the second-best tool by.
• A 0.9 to 0.1 win: is a confident pick. Route it.
• A 0.42 to 0.40 win: is a coin flip wearing a confident face and a fake ID. This is exactly how agents fire the wrong tool on ambiguous input, and gating on the margin is what catches it.
Real Ties, Not Softballs
The benchmark's ambiguous queries are genuine ties I found empirically, not hand-waved examples. "Convert this" scores identically for unit conversion and currency conversion. "Read this" scores identically for a plain file and a PDF. There is no right pick for either, only a right abstention.
The Number
Always-picking-the-top-tool fires a wrong tool on all 5 genuinely ambiguous queries in the 20-tool benchmark. Confidence-gated routing abstains on all 5, and it does that at zero cost: clear-query accuracy stays at 100% either way. Caution that costs nothing on the easy cases is the whole point.
100% Clear-Query Accuracy, on Queries Built From the Tools' Own Words
I checked where the benchmark's "clear" queries actually came from. They're close paraphrases of a tool's own description or keyword list, "what is the current share price of nvidia" for a tool whose keywords literally include "share price". That measures whether bag-of-words overlap can match a query to a tool built from the query's own vocabulary. It says nothing about routing a real question.
So I wrote fifteen natural questions the way a person actually asks them, "how much is 50 bucks worth in yen," "who's number does Sarah have on file," none built from a tool's own keyword list. Top-1 accuracy: 5 of 15. Every miss was an abstention, the router never once fired the wrong tool, but a router that shrugs at two-thirds of perfectly answerable questions has defeated the entire point of routing.
Underneath that I found a second, independent bug: the tokenizer regex splits on the apostrophe in a contraction, so "what's" becomes two tokens, "what" and a leftover "s". "what" gets dropped as a stopword, but the stray "s" survives as a real token and coincidentally ties against any other tool whose text happens to contain its own contraction. The bundled keyword lists have zero contractions in them, so this stayed completely invisible until natural phrasing, which is full of them, showed up.
The fix widens each tool's keyword list with the synonyms and phrasings a person actually uses, written from general domain knowledge before I looked at my own adversarial query set, and fixes the tokenizer to strip apostrophes instead of splitting on them. 33% jumped to 100% on that set, with zero regression on the original benchmark: clear-query accuracy stays at 100% and all 5 intentionally-ambiguous queries still correctly abstain. I built a second, held-out set of questions leaning on slang the fix wasn't built to anticipate, "rustle up a summary," "crunch these numbers", and evaluated it exactly once: the fix never turns a previously-correct answer wrong, but it doesn't fully generalize to slang outside what I enriched either, an honest limit of a bag-of-words approach with no embeddings behind it. The original registry is untouched; the fix ships as an explicit `registry_v2()`.
21 tests pass now, including a regression test that a contraction never produces a stray token again.