Renting a Moving Truck for One Bag of Groceries
I spent a good chunk of last year getting inference cost and p95 latency down on a SageMaker/vLLM endpoint serving tens of thousands of requests a day. The pattern that kept showing up: most teams solve "inference is expensive" by either always calling the big model and eating the cost, or always calling the small model and eating the quality hit on the requests that actually needed the big one. Neither is right, and it's the same mistake as renting a moving truck to pick up one bag of groceries.
The production fix there was model distillation plus batched serving, and it worked. That code was proprietary, so vllm-cost-router is the same shape of solution rebuilt from scratch as something anyone can run and poke at: route on complexity, cache what repeats, batch what's concurrent. None of those three ideas is novel on its own. Stacking all three in front of every request is what actually moves the number.
Technical Architecture
A FastAPI service, not a notebook you run once and forget about.
• Complexity-based routing: scores each prompt on length, code markers, and reasoning language, then sends it to a small or large model tier accordingly.
• Caching: a TTL cache keyed on prompt, model, and max_tokens returns repeated FAQs instantly instead of paying for them twice.
• Batching: the batch endpoint groups cache-misses by tier and dispatches each tier as one backend call, so fixed per-batch overhead amortizes across every request in it, the actual mechanism behind vLLM's continuous batching win.
• Production hardening: optional API-key auth, request size limits, a readiness probe, structured error handling, and a non-root Docker image with a health check.
Challenges & Solutions
The honest challenge wasn't the routing logic, it was making the benchmark trustworthy. A suspiciously round "94% savings!" is a red flag, and I wanted anyone cloning the repo to be able to check my work instead of taking the headline number on faith. The workload mix and the cost/latency gap between tiers are both defined in plain sight in the eval script, and the whole thing runs against a deterministic in-process mock so it needs zero API keys and zero cloud spend to reproduce.
That constraint (reproducible without any external dependency) forced a cleaner design than I would have landed on otherwise. It's pluggable to a real vLLM or OpenAI-compatible backend through environment variables when you're ready for production traffic.
Impact & Results
On a 500-request synthetic workload (25% genuinely complex, 25% repeated FAQs, 50% unique-but-simple templated requests), routing plus caching plus batching cuts total cost by 73% and p95 latency by 73%, with the same model quality on every request that actually needed the big model.
Full CI runs the test suite across Python 3.10, 3.11, and 3.13, plus a Docker build-and-smoke-test that boots the container and hits the live endpoint. All 19 tests green.