LLM Model Routing: How to Cut Cost Without Losing Quality

Two numbers dominate every article about LLM routing. RouteLLM (UC Berkeley + Anyscale, ICLR 2025) reports up to ~85% cost reduction while retaining ~95% of GPT-4 quality on MT-Bench, sending only ~26% of calls to the frontier model. FrugalGPT (Stanford, TMLR 2024) claims to match GPT-4 at up to 98% lower cost. Both results are real and both were honestly reported. They are also nearly useless as planning numbers, because they are properties of a specific dataset and a specific model pair, not of routing as a technique.

The thing worth internalizing before you build anything: routing can cut cost a lot, the savings are workload-specific, and the dominant risk is quality loss that is silent — it does not throw errors, it does not crash, and you will not notice it unless you instrument for it. Everything below is about earning the savings with measurement rather than assuming them.

Why routing pays at all: the 2026 price spread

Routing only makes economic sense because frontier and cheap models differ in price by one to two orders of magnitude, and most queries in a real workload do not need the frontier model. Current API pricing (per million tokens, in/out, June 2026) shows the spread plainly:

Model	Input	Output
Gemini 3.1 Flash-Lite	$0.10	$0.40
DeepSeek V3.2	$0.14	$0.28
Claude Haiku 4.5	$1.00	$5.00
Gemini 2.5 Pro	$1.25	—
GPT-5.4	$2.50	$15.00
Claude Sonnet 4.6	$3.00	$15.00
Claude Opus 4.7	$5.00	$25.00
GPT-5.5	$5.00	$30.00
GPT-5.4 Pro	$30.00	$180.00

(Source: CloudZero and TLDL LLM API pricing comparisons, June 2026.) The cheapest-to-frontier ratio is roughly 25–50x on input and similar on output, and note that output runs about 5x input across the Claude line. That ratio is the entire economic basis for routing. The savings model is simple:

savings ≈ (fraction of traffic served by the cheap model)
          × (price gap between tiers)
          − (router compute + latency cost)
          − (cost of escalations that were retried)

That last two terms matter more than most write-ups admit. If your base model is already cheap (Flash-Lite at $0.10/M), an LLM-as-classifier router that adds a whole extra inference call can eat the entire margin. The router only pays when the price gap it bridges is large relative to its own overhead.

The three strategies, precisely

People use “routing” and “cascading” interchangeably. They are different mechanisms with different failure profiles, and the difference is when the decision is made.

1. Difficulty-classification routing (pre-generation, one decision). A small, cheap classifier looks at the query and predicts which model to send it to, before any generation happens. BEST-Route uses a DeBERTa-small router; the vLLM Semantic Router uses a ModernBERT classifier to selectively send reasoning-heavy queries to a stronger model; IRT-Router applies Item Response Theory for an interpretable difficulty estimate. The decision should add under ~50ms. You pay the router cost once and never run the expensive model on queries the classifier deems easy. The risk: the classifier can be wrong, and you never get a second look — a misrouted hard query just gets a bad cheap answer.

2. Cascade (post-generation, sequential escalation). Run the cheap model, score its answer, and escalate to a stronger model only if the score is below a threshold. This is FrugalGPT’s core: a DistilBERT-based scoring function rates the cheap answer and escalates when confidence is low. AutoMix does the same with self-verification. The cascade is more cost-efficient than routing when the cheap model is usually right, because most queries terminate at tier one — but you pay for two generations on every escalation, and the whole scheme lives or dies on the quality of the scoring function (more on that below).

3. Preference / eval-gated routing. RouteLLM trains routers on human preference data (which of two model answers a person preferred) rather than on a hand-labeled difficulty signal. Of its four router types — matrix factorization, similarity-weighted ranking, a BERT classifier, and a causal-LLM classifier — the authors recommend matrix factorization (mf) as the most cost-effective, and report that routers trained on the GPT-4/Mixtral pair transfer to other model pairs without retraining. That transfer claim is the part to treat with suspicion; see the oracle-gap section.

The cleanest theoretical result here is from ETH Zurich’s SRI Lab (Dekoninck, Baader, Vechev — “A Unified Approach to Routing and Cascading for LLMs,” ICML 2025). They prove that pure routing and pure cascading are both special cases of one optimal policy, and that their cascade routing — which can both route up-front and escalate after seeing a response — consistently beats either approach alone, reporting 2–3x speedup at maintained accuracy. The practical lesson is not “use their code” (though it’s at github.com/eth-sri/cascade-routing); it’s that route-once and escalate-after are endpoints of a continuum, and the right design depends on whether you have a reliable post-generation confidence signal. If you do, cascading captures more of the gap. If you don’t, routing is safer because it never trusts a bad signal.

How much you actually save — the honest version

Start with the headline numbers and then immediately temper them. The papers report: RouteLLM ~85% cheaper at ~95% quality; FrugalGPT up to 98% cheaper (with reported savings ranging 50–98% across datasets); MixLLM (a contextual-bandit router) hitting 97.25% of GPT-4 quality at 24.18% of cost; R2-Reasoner (GRPO-trained) reporting 84.46% API cost savings. These are all real, all peer-reviewed or benchmarked, and all measured on distributions chosen by the authors.

Now the counterweight. RouterArena (arXiv 2510.00202, an 8,400-query open benchmark across 44 categories, 9 domains, and 3 difficulty levels) compared commercial and open-source routers head to head and found commercial routers do not necessarily win: GPT-5’s router ranked #7 (it’s limited to OpenAI models), and NotDiamond ranked #12 because it “frequently selects expensive models.” The best research routers in that study — vLLM-SR and CARROT — got roughly 35% cost cut at under 2% accuracy loss. That is far below the 80–98% vendor and headline numbers, and it’s the more honest planning figure for a new, well-built router on a workload you haven’t overfit to.

Worse, the fairness literature suggests even that may be optimistic out of distribution. Unified-evaluation work from 2026 (“Towards Fair and Comprehensive Evaluation of Routers,” arXiv 2602.11877; LLMRouterBench, arXiv 2601.07206) finds router performance is highly benchmark-dependent — routers do best when the eval distribution matches their training distribution, and under fair unified evaluation “most routing methods collapse to similar performance,” with several recent approaches failing to reliably beat simple baselines. So the FrugalGPT 98% and the RouteLLM 85% are not lies; they’re the top of a distribution-dependent range whose realistic middle, on your data, is closer to a third off.

The oracle gap, and why savings don’t transfer

Every router benchmarked above falls well short of the oracle — the hypothetical router that always picks the cheapest model that would get the answer right. The persistent gap to that oracle is the single most important fact about routing, and it’s widely ignored. The gap is not caused by insufficient model capability. It is caused by model-recall failure: the router cannot reliably identify the small set of queries that only the strong model can solve. That failure is systematic and it gets worse as difficulty rises — exactly the queries where misrouting is most expensive in quality terms.

This is also why the architecture of the router barely matters. The fairness work found that adding hidden layers to a router barely helps and increases overfitting; a linear probe is comparable or better. The bottleneck is the information content of your features and the match between train and serve distributions, not router capacity. And it’s why transfer claims should be doubted: a router that learned the GPT-4/Mixtral decision boundary on MT-Bench has learned a boundary specific to those models and that distribution. Treat the oracle gap as a hard ceiling on your savings, and assume the published transfer is the best case, not the expected case.

The confidence-signal problem: why naive cascades silently fail

Here is the failure mode that makes a cascade look cheap while quality quietly drops. A cascade escalates when the cheap model’s confidence is low. The obvious confidence signal is the model’s own self-reported confidence — “I’m 95% sure.” That signal is poorly calibrated. LLMs are frequently confidently wrong and unconfidently right, so a naive cascade ends up escalating easy queries the cheap model got right (wasting money) while keeping wrong cheap answers on hard queries it was falsely confident about (losing quality). It optimizes the cost number and silently degrades the quality number, which is the worst of both.

GATEKEEPER (arXiv 2502.19335, ICML/NeurIPS 2025) attacks this directly: it’s a loss function that fine-tunes the small model to output high confidence when it’s actually correct and low confidence when it’s wrong, with a trade-off parameter α controlling cautiousness versus sharpness. The broader finding it confirms — also seen in probe/hidden-state confidence work — is that raw self-reported confidence is a bad deferral signal, and a probe on the model’s internal states beats it. If you build a cascade, the scoring function is not a detail you bolt on at the end. It is the system. A cascade with an uncalibrated confidence signal is not a cheaper version of the strong model; it’s a more expensive version of the weak model that occasionally costs frontier money.

Failure modes catalog

• Silent quality degradation. Routing to cheaper models fails quietly. In multi-agent benchmarks (Agent Drift, arXiv 2601.04170), about 16% of scenarios produce a silent failure where the system proceeds with no error or warning. You don’t get a stack trace; you get a slightly worse answer that looks fine.
• Cumulative/slow drift. Regressions are often cumulative — a PSI- or divergence-style metric inches up daily without ever breaching a single-interval alert. As the practitioner saying goes, drift monitors miss the slowest failures. A 0.3% daily quality erosion is invisible to a same-day alert and catastrophic over a quarter.
• Distribution overfit. A router tuned to a benchmark beats baselines on that benchmark and collapses on real traffic whose mix of intents, languages, and difficulty differs.
• Model-recall failure on hard queries. Already covered — the oracle gap, concentrated where it hurts most.
• Router cost/latency eats the savings. An LLM-as-classifier adds a full inference round-trip. On a cheap-base workload the router can cost more than it saves. Do the break-even math: if the router call costs more than (escalation rate avoided) × (price gap), you’re losing money to route.
• Misattributed root cause. When quality drops, teams blame “model drift.” The actual cause is frequently a logging gap, a stale feature, a threshold mistake, or the routing change itself. Routing changes are a leading, under-suspected cause of regressions precisely because nobody instruments them as deploys.

Instrumenting a router to prove it isn’t degrading

This is the part top-ranking content hand-waves as “monitor quality.” Here is the actual pipeline. The premise: a routing change is a code deploy with a quality blast radius, and you ship it like one.

1. Pre-merge eval gate. Before any routing change merges, run it against 50–500 representative cases with rubric-based scoring. No change to thresholds, classifiers, or model assignments ships without passing. This catches the gross regressions cheaply and is the floor, not the ceiling, of safety.

2. Shadow mode. Score both old and new routing on the same live inputs without serving the new path to users. Promote only if the production delta stays inside the offline confidence interval you measured in the gate. The discipline that makes shadow mode worth anything: it must run automated eval on the shadowed outputs. Shadow mode without scoring is just expensive logging — you’ve recorded what the new router would have done and learned nothing about whether it was better.

3. Canary. Route 1% of real traffic through the new path, run the eval pyramid in real time, and gate wider rollout on rubric scores staying within baseline bounds. The canary exists to catch distribution effects the shadow set missed.

4. Eval-gated auto-rollback. Compute a rolling per-rubric pass rate on a ~15-minute window and automatically revert if any rubric drops below roughly 1.5x its noise floor versus the incumbent. The noise floor matters: you measure the natural variation of each rubric on the incumbent first, then trip on a multiple of it, so you’re not rolling back on sampling jitter.

Metrics to log on every request, not just averages:

• Cost per request (and per intent class)
• Output-length distribution — a cheap model padding or truncating shows up here before it shows up in a rubric
• Refusal rate — cheap models refuse more; a rising refusal rate is silent degradation
• Regeneration / retry rate — users or downstream agents retrying is a quality signal that bypasses your rubric
• Escalation rate — in a cascade, this is your cost dial and a leading indicator: a falling escalation rate with flat quality is good; a falling escalation rate is sometimes a miscalibrated scorer keeping more bad cheap answers

The non-negotiable plumbing requirement: keep request IDs, traces, routing decisions, and scores joined. When a quality metric moves, you must be able to attribute the drop to a specific routing change — which model served which query and why. Without joined traces you can see that quality dropped and have no way to prove the router did it, which is how routing changes get blamed on model drift.

Tooling: build vs buy

The fork that matters is quality-aware ML routing vs simple cost/fallback routing.

• Simple cost/fallback gateways: OpenRouter (managed, 200+ models, one key, automatic fallbacks); LiteLLM (open-source, 100+ providers, OpenAI-compatible, per-key budgets, free to self-host); Portkey (went fully Apache-2.0 in March 2026 — 1,600+ models, 40+ guardrails, conditional routing, circuit breakers, semantic caching); Vercel AI Gateway. These give you provider abstraction, fallbacks on failure, and rule-based routing. They do not decide which model is right for this query on quality grounds.
• Quality-aware ML routers: NotDiamond (optimizes for accuracy on your own data — but ranked low on cost in RouterArena because it leans expensive) and Martian (adaptive, learns from your traffic). These make the hard prediction, with all the oracle-gap caveats above.

For the eval layer, Braintrust and FutureAGI-style observability cover the scoring, shadow, and canary machinery. A managed router is enough when your spread is modest, your traffic is forgiving, and you mainly want fallbacks and a single key. You need your own classifier plus the eval gate when the price gap is large enough to justify the engineering, your distribution is specific enough that off-the-shelf routers overfit against you, and a silent quality drop is expensive.

A pragmatic recipe

Start with a two-tier setup: a cheap default and a frontier fallback. Don’t build a five-model router on day one — you can’t even instrument it. Gate every routing change behind the full pipeline (eval gate → shadow → canary → auto-rollback); routing changes that skip the gate are how silent degradation enters. Prefer a calibrated classifier or a tuned-confidence cascade over an LLM-as-judge router, for latency and cost. Measure savings on your distribution, not the benchmark’s — the published 85% and 98% are ceilings under ideal conditions; budget closer to the RouterArena ~35%-at-<2%-loss range and be pleased if you beat it. Treat the oracle gap as your ceiling and model-recall on hard queries as your dominant risk.

Routing is a real cost lever — the 25–50x price spread is genuine and most of your traffic genuinely doesn’t need the frontier model. But it is a lever you earn with measurement, not a free 80% discount you switch on. The cheap version of this is also the dangerous version, because the failure is quiet.

Primary references

RouteLLM (arXiv 2406.18665, ICLR 2025) · FrugalGPT (arXiv 2305.05176, TMLR 2024) · Unified Routing & Cascading / cascade routing (arXiv 2410.10347, ICML 2025) · GATEKEEPER (arXiv 2502.19335) · RouterArena (arXiv 2510.00202) · Dynamic Model Routing and Cascading survey (arXiv 2603.04445) · Towards Fair and Comprehensive Evaluation of Routers (arXiv 2602.11877) · LLMRouterBench (arXiv 2601.07206) · RouterBench, RouterEval · Agent Drift (arXiv 2601.04170) · CloudZero / TLDL LLM API pricing comparisons (June 2026).