You Are Renting a Moving Truck to Carry One Bag of Groceries
Almost everyone overpays for inference the exact same way. They fire every request at the biggest model they own, then act shocked when the bill reads like a car payment. The other camp does the reverse. They route everything to the cheap model and quietly ship worse answers on the requests that actually mattered.
Both are wrong. Both are lazy. And both are fixable in an afternoon.
I spent most of last year cutting inference cost on a vLLM endpoint doing tens of thousands of requests a day. There was no clever prompt that saved the day. It was three boring ideas stacked on top of each other: route on complexity, cache what repeats, batch what runs at the same time. That code lived behind an NDA. This is the same idea rebuilt in the open, small enough to read in one sitting.
How It Actually Works
A FastAPI service. Not a notebook you run once and forget in a tab.
• Routing: it scores every prompt on length, code markers, and reasoning language, then sends it to the small tier or the big tier. "What are your hours?" does not need your smartest model. It stops pretending otherwise.
• Caching: a TTL cache keyed on the prompt, model, and max_tokens hands back repeat questions instantly. You pay for an answer once, not every time someone asks it again.
• Batching: the batch endpoint groups cache-misses by tier and fires each tier as a single call, so the fixed overhead spreads across every request in the group. That is the real engine behind vLLM's continuous batching win.
• Hardening: optional API-key auth, request size limits, a readiness probe, clean error handling, and a non-root Docker image with a health check. Boring on purpose.
The Part Nobody Wants to Talk About
The routing logic was never the hard part. Making the benchmark honest was. Any repo that promises you a suspiciously round "94% savings" is selling something. So I put the workload mix and the cost gap between tiers in plain sight in the eval script, and the whole thing runs against a deterministic mock. Zero API keys, zero cloud spend, and you can check my math instead of trusting it.
That one constraint, reproducible with nothing external, forced a cleaner design than I would have shipped otherwise. Point it at a real vLLM or any OpenAI-compatible backend with an env var when you are ready for real traffic.
The Number
On a 500-request workload (25% genuinely hard, 25% repeat FAQs, 50% simple templated stuff), stacking routing, caching, and batching cuts total cost by 73% and p95 latency by 73%.
Then I ran the same router against real weights, because a mock will tell you whatever you designed it to say. Two real Qwen models as the tiers, real wall-clock latency, real per-token pricing. The saving fell to 15%, because the long analytical answers that correctly stay on the big model dominate token spend. And the 0.5B tier scored 79% on exactly checkable prompts (it thinks 45 plus 55 is 90), so the cost win comes out of your quality budget when your tiers are that far apart. Both findings are in the README, because they are more useful than the flattering version.
I Was Measuring the Wrong Thing Entirely
For a while this page said "same answer quality on every request that actually needed the big brain." I had measured cost carefully, disclosed the methodology, and run it against real weights. What I had never measured, at all, was whether the router sent the right requests to the right tier. Cost and correctness are different questions, and only one of them was on the dashboard. Route everything to the cheap tier and you save 50%; no cost graph will ever tell you that was a mistake.
So I built a labeled corpus and measured it. The router misrouted 100% of held-out complex requests to the cheap model. Not most. All of them.
The cause is embarrassing and, I suspect, common. The router scored complexity by searching for words: "explain", "analyze", "compare", "design". The four complex prompts in my own benchmark each opened with one of those words. The benchmark was a mirror. It confirmed the router could find the words the router was looking for, and could not possibly have detected one that fails on anything phrased differently.
Write the same difficulty the way an engineer actually types it, and it falls apart completely. "Our checkout drops 4% of orders when Redis fails over" scores 0.072. "The p99 on our embedding endpoint tripled after we doubled batch size" scores 0.058. The threshold is 0.50. These aren't near misses; they're a router that has no idea what it's looking at.
The worst part: the evidence was already sitting in my own published results. The real-model benchmark showed the router scoring 19/24, exactly identical to always-using-the-small-model, while costing 13x more than it. A router that adds zero quality over the cheap tier isn't routing. I'd written that table, published it, and not read it.
What Replaced It, and What It Still Gets Wrong
The new scorer reads structure instead of vocabulary. Does the request name actual subsystems, Postgres and replication and deadlocks? Does it carry quantities and units? Does it weigh named alternatives, or embed code? And the correction that mattered most: is it about the caller's own account, which is what makes "why was I charged twice" a database lookup wearing a reasoning word rather than an engineering problem.
I hand-fitted those weights against my corpus, which means scores on that corpus prove nothing. So I wrote a held-out set afterwards with the weights frozen and ran it exactly once: 0.955 accuracy, one miss in ten, against the old router's zero for ten. That single number is the only one on the page I'd defend, because it's the only one measured on data the weights never saw.
I've deliberately left that one miss unfixed. It's "two writers both read-modify-write the same counter and we lose about 3% of increments": a real concurrency bug described without naming a single recognizable subsystem. I could patch the term list to catch it in about a minute, but tuning against a holdout after reading the result is how a holdout quietly stops being one. It stands in the README as the documented failure mode, which is that the new router leans on domain vocabulary and a hard problem in entirely plain language can still slip past it.
The tradeoff is now reported honestly, and it does not flatter the fix. On realistic traffic the broken router looks better on every number the cost benchmark reports: 72% savings against 71%, and a materially better p95. It wins precisely because it's answering hard questions with the small model. One point of cost is what correctness costs here, and reporting only the cost column would have hidden that in either direction. CI now fails the build on a misroute regression, because quality can regress silently in a way cost never does. 65 tests, all green.