The Cure for the Intern Who Never Says "I Don't Know"
You have met this person. Every answer lands with total conviction, right or catastrophically wrong, delivered at the same volume. An auto-tagger works exactly the same way. It is wrong precisely as often as it is overconfident, and it will never once warn you which mode it is in.
A "smart watch band" is genuinely both electronics and apparel. Forcing a confident guess there is not intelligence, it is a coin flip wearing a suit. So taggate does the one thing the intern never does. It admits when it is not sure and passes that item to a human.
It scores confidence as the margin between the top category and the runner-up, not just whether it found a match at all, and anything below the line goes to a person instead of getting silently mislabeled.
Built to Know Its Own Limits
TypeScript, zero runtime dependencies.
• Margin confidence: winning a category 1-to-0 with no competition is not the same as squeaking it 5-to-4, and the score knows the difference. A contested item cannot slip through on one lucky keyword.
• The gate: clears the threshold, auto-tag it. Falls short, hand it to a pluggable reviewer.
• Bring your own brain: the keyword tagger swaps for a real LLM behind the same interface, and the gating logic does not change.
• Bring your own human: the test reviewer swaps for a Slack approval or a ticket queue in production.
The One Decision That Mattered
The whole thing hinges on rejecting the naive signal. Early versions treated any keyword hit as confidence, so a product with one accidental match for the wrong category got auto-tagged with a straight face. Scoring the gap between first and second place, not the raw strength of first place, is the entire reason the ambiguous cases get caught.
And I refused to benchmark it on an easy catalog. 12 products, 4 of them deliberately ambiguous, because a test that never challenges the gate proves nothing about the gate.
The Number
Auto-tag everything and you hit 75%, getting the ambiguous quarter wrong at full confidence every time. Let the gate escalate the hard 42% to a human and it reaches 100%.
The Escalation Rate Was Suspicious
42% is a strange number for a "smart" gate to land on. So I asked a more basic question: what happens when this thing sees a product it wasn't built around? I ran it against 24 ordinary listings, a 4K monitor, a wool coat, an espresso machine, none deliberately tricky. It escalated 22 of them.
Not because they were hard. The electronics keyword list was ["battery", "charger", "bluetooth", "watch", "smart", "screen", "wireless", "usb"]. Eight words. None of them are "monitor," "laptop," "headphone," or "speaker." Every clean item in the 12-product demo catalog happens to hit 3 or 4 of its category's 8 keywords, because the keyword lists were written by reading the catalog. I'd built a benchmark that measured whether the tagger could find the words I'd handed it, and called that "confidence."
A system that escalates 92% of a catalog isn't safely deferring to a human. It's a human doing the job with extra steps in the way.
Bigger Keyword List, Worse Confidence
The fix seemed obvious: quadruple the vocabulary, built from general product knowledge instead of the demo catalog, plus basic stemming so "Rechargeable Batteries" and "Charging Cable" stop scoring zero on electronics because neither contains the literal substring "charger."
First run: 0% auto rate on the original catalog. Worse than doing nothing. The scoring was hits / list.length, so a category with 45 keywords scores a hit at 1/45 instead of 1/8, and every confidence number collapsed. A bigger, more useful taxonomy was punishing itself for being bigger. Confidence comes from the raw hit count now, not a fraction of an arbitrarily-sized list.
Then It Broke a Tie the Wrong Way
Fixing that pushed coverage up, and introduced a new, quieter failure: one of the demo catalog's four deliberately-ambiguous items, a "Kitchen Storage Folder Organizer," crossed the confidence line into a wrong auto-tag. The expanded office list now matched "organizer," "folder," "paper," and the literal word "office," pulling it decisively away from kitchen. Broader recall cuts both ways: it also strengthens whichever side of a genuine tie happens to gain more keywords.
Fixed by raising the threshold to 0.55, tuned against the original catalog and the 24 adversarial listings only, before I'd looked at a single number from a holdout set. I'd actually swept the threshold once already with the holdout numbers sitting right there in the same output, which meant that holdout was compromised the moment I read it. I discarded it and wrote a fresh one.
Held out and run exactly once, on 15 more ordinary listings written after the taxonomy, stemmer, and threshold were frozen: auto rate 7% to 67%, zero wrong auto-tags, same as the other two catalogs. 29 tests now, and CI fails the build on a wrong auto-tag or a coverage regression.