LLM Routing: How It Works, Strategies and Real Data

What is LLM routing?

LLM routing is the practice of sending each request to the model best suited to it instead of one default model. A router reads the request, estimates how hard it is, and picks the cheapest model likely to answer well, balancing cost, quality and latency.

How much routing saves depends on how many of your requests are simple. On 2024 benchmarks, the RouteLLM team cut costs by 35% to over 85% while reaching 95% of GPT-4's performance (details below). For a workload with many simple requests we estimate a 40 to 70 percent lower bill, but that is an estimate, not a measurement: the one live account we have measured saw its bill drop about 11%, because most of its traffic is long writing that stays on the stronger model. Done poorly, routing causes quality drops that are hard to trace. This guide covers the strategies, what the research found, how to evaluate a router, and the mistakes to avoid.

---

Why routing exists

Model prices now span two orders of magnitude, often within one vendor. Standard prices per million tokens on October 15, 2026, from the OpenAI (short-context rates), Anthropic and Google pricing pages:

Tier	Model	Input	Output
Top	GPT-6 Astra	$10.00	$50.00
Top	Claude Fable 5.1	$10.00	$50.00
Middle	Claude Opus 5.5	$4.00	$20.00
Middle	GPT-6.1 Sol	$2.00	$10.00
Middle	Claude Sonnet 5.5	$2.00	$10.00
Middle	Gemini 3.1 Pro Preview (prompts up to 200k tokens)	$2.00	$12.00
Low-cost	Gemini 3.5 Flash-Lite	$0.30	$2.50
Low-cost	GPT-6 Luna	$0.10	$0.50
Low-cost	Claude Haiku 5.5 (prompts up to 100,000 tokens)	$0.10	$0.50

GPT-6 Astra costs 100 times as much per token as GPT-6 Luna ($10 / $0.10 for input, $50 / $0.50 for output), and Claude Fable 5.1 costs 100 times as much as Claude Haiku 5.5. The vendors position them differently too: OpenAI's models page suggests Astra for complex reasoning and coding and Luna for cost-sensitive, high-volume workloads.

On many requests the quality gap is far smaller than the price gap, but not on all of them. RouterBench found that a perfect router would rarely have picked GPT-4, and our own checks found cheap models that fell short on research-style work (both below). Routing exists to pay top-tier prices only for the requests that need them.

---

Router, gateway, proxy: the terms, defined

These words get used interchangeably, so here is the plain version.

An AI router decides, per request, which model should answer. It reads the request, judges how hard it is, and sends it to the cheapest model that can handle it well. Routing is the decision.

An LLM gateway is the single entry point your app sends every AI call through. It forwards the call to the right provider and adds the things you would otherwise build yourself, such as routing, cost tracking, rate limiting, caching and failover. The gateway is the layer; routing is one feature inside it.

An AI inference proxy is how a gateway usually works in practice. Your app points its base URL at the proxy instead of at OpenAI, the proxy forwards the request with your own provider key, and it adds routing and tracking on the way through. Besides the base URL and a key for the proxy, your code stays as it is. The proxy is the plumbing.

In short: the proxy is where the call travels, the gateway is the set of features it passes through, and the router is the decision that picks the model. PromptUnit is a router you reach by pointing your base URL at it: it switches each request to a cheaper model when one is just as good, and its dashboard shows what each request cost. It does not cover failover: it does not move your requests to another provider when one is down.

---

LLM routing strategies compared

Here are six common approaches. They differ in what data they need before they work and in how much time they add to each request.

Strategy	Data needed	Added latency	Best for
Rules	None: you write the mapping	None (an if-statement in your code)	A few well-understood features
Classifier	Signals you choose, or labeled easy and hard requests	Under a millisecond for heuristics; one short model call for a small LLM classifier	Mixed traffic on one endpoint
Preference-trained router	Pairs of answers marked better or worse, plus some examples from your own domain	One fast router pass before a single model call	Choosing between one strong and one cheap model
Cascade	A scoring function or judge that decides if an answer is good enough	A full extra model call each time a request moves up	Answers that can be checked; work where cost matters more than speed
Semantic (embedding) routing	Example prompts for each route	One embedding call	Sending known intents to known models
Latency or fallback routing	Live latency and error data per provider	None when the first call works; retry time when it fails	Uptime and speed, not cost

For scale, LiteLLM's Auto Routing docs list sub-millisecond routing for heuristic or keyword rules and about 100 to 500 ms for the embedding call of its older semantic router. Real routers often combine several of these: rules for known features, a classifier for the rest, and a fallback when a provider errors.

1. Rules

The simplest approach: you write explicit routing rules based on feature, endpoint, or request metadata.

def route_request(feature: str, messages: list) -> str:
    # Static routing by feature
    if feature in ["code_review", "architecture_analysis"]:
        return "gpt-6-astra"
    elif feature in ["summarization", "classification", "extraction"]:
        return "gpt-6-luna"
    else:
        return "gpt-6.1-sol"  # default

Advantages: Predictable, auditable, no latency overhead.

Disadvantages: Brittle. Feature tags don't always reflect actual complexity. A "summarization" feature might sometimes receive complex multi-document requests. New features start unrouted by default. Rules accumulate and become hard to maintain.

Static rules are a reasonable starting point for small teams with well-understood traffic. They break down as the application grows.

2. Classifier (content-based)

The proxy, or a classifier in your stack, reads the actual request and decides based on complexity signals:

• Token count of the input
• Presence of code, structured data, or domain-specific terminology
• Instruction complexity (single vs. multi-step)
• Context depth (multi-turn conversation length)
• Explicit complexity markers in the prompt

def classify_complexity(messages: list) -> str:
    total_tokens = estimate_tokens(messages)
    has_code = any("```" in m.get("content", "") for m in messages)
    is_multistep = count_instructions(messages) > 3
    is_long_context = total_tokens > 8000

    if has_code and is_multistep:
        return "gpt-6-astra"
    elif is_long_context:
        return "gpt-6.1-sol"
    elif total_tokens < 2000 and not is_multistep:
        return "gpt-6-luna"
    else:
        return "gpt-6.1-sol"  # safe default for ambiguous cases

Advantages: Adapts to actual request content, not just feature labels. Works for mixed traffic on the same endpoint.

Disadvantages: Classifier accuracy matters: wrong calls on edge cases lower quality. It needs upkeep as your traffic changes.

3. Preference-trained router

A learned router trained on pairs of answers: for each prompt, which of two models gave the better response, or a tie. RouteLLM (below) trained its routers this way on public Chatbot Arena data, then picked between a strong and a weak model for each request. Its paper measured the overhead: the matrix factorization router handled about 155 requests per second on a single cloud VM, at $3.32 per million requests.

Advantages: Learns which kinds of prompts need the strong model. Only one model call per request.

Disadvantages: Needs preference data that resembles your traffic. RouteLLM's routers trained only on Arena data were close to random on MMLU until the team added examples from that domain.

4. Cascade

Try the cheapest model first, score its answer, and move up only if the score is too low. FrugalGPT (below) is the best-known example.

def cascade(messages: list, threshold: float = 0.9) -> str:
    """Try models from cheapest to strongest; stop when an answer scores well enough."""
    ladder = ["gpt-6-luna", "gpt-6.1-sol", "gpt-6-astra"]
    for model in ladder[:-1]:
        answer = call_model(model, messages)
        if score_answer(messages, answer) >= threshold:  # your judge or checker
            return answer
    return call_model(ladder[-1], messages)

Advantages: You pay for the strong model only when the cheap answer fails the check.

Disadvantages: Escalated requests pay for two or more calls and wait for each. The result is only as good as the scorer: RouterBench found that cascade performance can drop quickly once the judge's error rate passes 0.2. The RouteLLM paper notes that methods relying on multiple model queries can increase latency.

5. Semantic (embedding) routing

Embed the incoming prompt, compare it with example prompts for each route, and send it to the model assigned to the closest route.

Advantages: Easy to set up for known intents (billing questions, code, translation), with no training run.

Disadvantages: One embedding call per request. It matches topic, not difficulty, so a hard billing question goes to the same model as an easy one. LiteLLM now lists its embedding-based semantic router as deprecated in favor of its Auto Routing feature.

6. Latency or fallback routing

Picks among providers or deployments by measured speed or health, or retries elsewhere when a call fails. It does not lower cost by itself: it is a reliability and speed tool. LiteLLM, for example, documents latency-based routing and provider fallbacks. For the details of building failover, see our provider failover guide.

---

What the research on LLM routing shows

Three public studies are the usual starting point. All three use 2023 and 2024 models on public benchmarks, so read them as evidence that routing can work, not as a forecast for your traffic.

RouterBench (March 2024)

RouterBench is a benchmark for comparing routers, from researchers at Martian (a company that sells a router), UC Berkeley and UC San Diego (arXiv 2403.12031). It holds 405,467 answers from 11 models across 8 datasets, including MMLU, MT-Bench, GSM8K and MBPP, plus a retrieval (RAG) set. Three findings matter in practice:

• A perfect router would rarely pick the most expensive model. The paper's Oracle picks the best-performing model for each question, and the cheapest one when several tie. On MMLU it scored 0.957 at a total cost of 0.297, against 0.828 at 4.086 for GPT-4 alone, which is better accuracy at about 7% of the cost. GPT-4 was seldom chosen. The Oracle knows the answers in advance, so it is a ceiling, not something you can deploy.
• Real routers were not much better than a simple mix. KNN and MLP routers matched the best single models at lower or similar cost, but none clearly beat the Zero router, a baseline that mixes models in fixed proportions for each budget without looking at the request.
• Cascades depend on the judge. With a judge that was wrong up to 10% of the time, cascades beat every single model and the Zero router by a significant margin. Past a 20% error rate, their performance could drop quickly.

The paper also found that the cost of comparable performance routinely varies by 2 to 5 times between LLM services.

RouteLLM (July 2024)

RouteLLM comes from LMSYS, with researchers from UC Berkeley, Anyscale and Canva (blog post, July 1, 2024; paper). The team trained four routers on public Chatbot Arena preference data: similarity-weighted ranking, matrix factorization, a BERT classifier and a causal LLM classifier. Each router picks between a strong model (GPT-4 Turbo) and a weak one (Mixtral 8x7B) for every request.

• Savings: cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K compared with using only GPT-4, while still reaching 95% of GPT-4's performance.
• Data matters: trained only on Arena data, all routers performed near random on MMLU, whose questions were mostly out of distribution. Adding about 1,500 MMLU examples, under 2% of the training data, brought the best router to 95% of GPT-4's performance with 54% of calls going to GPT-4.
• Transfer: on MT Bench, the same routers kept comparable results when the pair was swapped to Claude 3 Opus and Llama 3 8B, without retraining.
• Versus paid routers: in the team's own MT Bench comparison, its routers matched Martian and Unify AI while being over 40% cheaper.

The code and datasets are open source under Apache 2.0 at github.com/lm-sys/RouteLLM.

FrugalGPT (May 2023)

FrugalGPT, from Lingjiao Chen, Matei Zaharia and James Zou at Stanford (arXiv 2305.05176), is the best-known cascade. It sends a query to a cheap model first, has a small scoring model rate the answer, and moves to the next model only when the score is too low. The study used 12 LLM APIs from 5 providers, at March 2023 prices, on three datasets: HEADLINES (financial news), OVERRULING (legal) and COQA (reading comprehension).

• The abstract reports that FrugalGPT "can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost."
• In the HEADLINES case study, the learned cascade tried GPT-J, then J1-L, then GPT-4, and accepted GPT-J's answer when its score was above 0.96. With a budget of one fifth of GPT-4's cost, it cut cost by 80% and raised accuracy by 1.5% compared with GPT-4.

These tasks have short answers that can be checked against a label, which makes scoring easy. Open-ended answers are harder to score, which is where RouterBench's finding about judge errors matters.

What the three agree on

• Cheap models answer a large share of benchmark questions well enough.
• The router or judge is the weak point: a poor one wipes out the gain.
• Results depend on the data. The same RouteLLM routers saved over 85% on MT Bench and 35% on GSM8K.

Our own early data

We can measure part of this on real traffic. With PromptUnit's optional side-by-side checks, cheaper models answer a sample of real requests and the model the app asked for judges whether each cheaper answer is as good. On one production account, between October 5 and 7, 2026:

• A cheaper model matched or beat the chosen model in 18 of 24 comparisons (about 75%).
• On those calls, the cheaper models cost roughly 89% less.
• The result depends on the pair: Claude Sonnet 5 to Gemini 2.5 Flash matched 9 of 10, Sonnet 5 to GPT-5.4 mini matched 8 of 10, and swaps below Claude Haiku 4.5 matched 1 of 4.

This is a small, early sample from one account: 24 comparisons, against RouterBench's 405,467 answers. The judge was Claude Sonnet 5, the model the app asked for, so it is not an independent evaluation. Its value is that it comes from real requests rather than a benchmark. The full method and limits are in What We Measured Routing GPT-4o Traffic to Cheaper Models.

---

How to evaluate an LLM router

Benchmarks show that routing can work. Whether a given router works on your traffic is a separate question, and the papers above show how to answer it.

1. Define quality per task

Quality means different things for different tasks:

• Classification: Accuracy rate (verifiable, has ground truth)
• Summarization: Human preference scores, factual accuracy, key point coverage
• Code generation: Test pass rate, syntactic correctness, reviewer acceptance
• Customer support: Resolution rate, customer satisfaction scores, escalation rate
• Creative generation: Human preference (subjective, difficult to automate)

Tasks with ground truth (classification, extraction, structured output) are easy to check at scale. Tasks with subjective quality need human review or a model judge, and a judge has its own error rate.

2. Draw the cost-quality curve

Run the router at several thresholds and plot quality against cost, or against the share of requests sent to the strong model. Then draw the same curve for a baseline: a random router that sends a fixed share of requests to the strong model (RouteLLM's baseline), or RouterBench's Zero router. A router is worth its complexity only if its curve sits above the baseline's.

3. Summarize with one number

The two papers define metrics you can borrow:

• PGR and APGR (RouteLLM): PGR is the share of the quality gap between the weak and the strong model that the router recovers: (router score minus weak score) divided by (strong score minus weak score). APGR averages PGR across cost levels, from 0% to 100% of calls going to the strong model, so a router cannot score well just by always calling the expensive model.
• CPT(x%) (RouteLLM): the smallest share of calls to the strong model needed to recover x% of the gap. Lower is cheaper.
• AIQ (RouterBench): the area under the router's cost-quality curve, divided by the cost range. Higher is better.

4. Test on your own traffic before you switch

Before routing live traffic, run both models in parallel on a sample:

import asyncio
import random
from datetime import datetime, timezone

async def shadow_test(messages: list, sample_rate: float = 0.05):
    """Send a small share of traffic to both models and log the pair for review."""
    if random.random() > sample_rate:
        return None  # not sampled

    strong, cheap = await asyncio.gather(
        call_model("gpt-6.1-sol", messages),
        call_model("gpt-6-luna", messages),
    )
    log_comparison(
        messages=messages,
        strong_response=strong,
        cheap_response=cheap,
        timestamp=datetime.now(timezone.utc),
    )
    return strong, cheap

Shadow test results build a comparison dataset you can score against your quality rubric. That tells you, with data, whether routing holds up on your traffic before it affects any user.

5. Check the judge

If a model grades the answers, its mistakes become your router's mistakes: RouterBench's cascades lost their edge as judge error rose past 20%. Spot-check a sample of the judge's verdicts by hand, and do not read "no errors" as "good answers."

An example from an app the PromptUnit team runs itself (checks up to October 12, 2026): Claude Haiku 4.5 served its web-research step for five days with no errors and answers that passed the automatic format checks, yet it was judged worse than Claude Sonnet 5 in 2 of 3 checks, so PromptUnit stopped moving that step and kept it on Sonnet 5. Claude Haiku 5.5 was judged as good in only 4 of 8 checks on the same step. On the app's video-prompt step, GPT-5.4 mini was judged as good in 23 of 23 checks at about 73% lower cost. These are small samples, and the judge was Claude Sonnet 5, the model the app asked for, so they are not independent.

6. Keep monitoring after launch

Routing changes the model mix. As it does, watch for quality drops:

• Signals to watch: User corrections, retry rates, explicit negative feedback, task-level downstream metrics (conversion rate on content generated by each model)
• Drift: If your traffic shifts (new features, new user segments), routing rules that held before may need revisiting
• Per-model quality scores: Keep rolling quality metrics by model for each task category

---

The routing decision architecture

For production systems, the routing decision needs to be fast, reliable, and auditable.

Incoming Request
      ↓
Feature Extraction
  - Token count
  - Task type signals
  - Code/data detection
  - Multi-step instruction count
  - Context depth
      ↓
Routing Classifier
  - Rule-based threshold checks
  - Learned complexity score (if applicable)
  - Override rules (always the strong model for feature X)
      ↓
Model Selection
  - Primary model
  - Fallback model (if primary is rate-limited or down)
      ↓
Request Forwarding
      ↓
Response + Routing Metadata
  - Which model was used
  - Why it was selected
  - Cost of this call
  - Quality confidence

Every routing decision should be logged with enough information to audit it later. "Why did this request go to gpt-6-luna?" should always be answerable from the logs.

---

Cross-provider routing: the next step up

Routing between a flagship and its own small model is the most common starting point. Cross-provider routing extends the choice across vendors, because the best model for a task is not always the cheapest one from the vendor you happen to use:

Use case	Models to compare (October 2026)	What to compare
Complex reasoning	GPT-6 Astra, Claude Fable 5.1, Claude Opus 5.5, Gemini 3.1 Pro Preview	Quality on your hardest requests
Routine, high-volume work	GPT-6 Luna, Claude Haiku 5.5, Gemini 3.5 Flash-Lite, DeepSeek Flash	Price per request at acceptable quality
Code generation	GPT-6 Astra, Claude Opus 5.5, Claude Sonnet 5.5, DeepSeek V4 Pro	Test pass rate per dollar
Long context	Gemini 3.8 Flash (1,048,576 input tokens), Claude Sonnet 5.5 (1M tokens)	Context length and long-prompt pricing

Context windows are from Google's Gemini 3.8 Flash page and Anthropic's models overview; DeepSeek Flash and V4 Pro are listed on DeepSeek's pricing page.

Different vendors price their tiers differently, and the strongest model on a given task is not always the most expensive. Routing per task across vendors uses both facts. The same spread shows up outside chat, for example in audio transcription (the numbers). PromptUnit routes chat requests only, so that choice is yours.

Our clearest result so far is cross-vendor: see Our own early data, above.

Cross-provider routing can also add resilience, but only if the router is built to fail over when a vendor is down. PromptUnit is not: if a cheaper model it chose fails, it retries on the model you asked for, and it does not reroute requests to another provider when yours is down. To build provider failover yourself, see our provider failover guide. Routing across vendors is also a hedge against price changes: when one vendor raises prices, a router that picks by price can shift traffic to the others without a code change.

The catch is integration: every provider has a different API, and a cheaper vendor only helps if its answer is actually as good, so each pair has to be quality-checked before it is trusted. A proxy helps here: your application sends one format to one endpoint, and the proxy translates for each provider it supports.

---

Build it yourself or use a proxy

Building routing from scratch involves:

1. A classifier (rules or a learned model)
2. A model registry (which models are available, at what cost, with what rate limits)
3. A routing decision engine
4. Fallback and retry logic for each provider
5. Quality monitoring and alerting
6. A logging and cost attribution layer
7. A configuration system so rules can be updated without code deploys

That is a real project with ongoing upkeep, since model names and prices change often, as the tables above show. It can make sense at a company with heavy LLM traffic and an infrastructure team. Open-source projects such as RouteLLM and LiteLLM cover parts of it (see the FAQ below).

The other option is a hosted router. PromptUnit is one: it checks each request and uses a cheaper model when one is just as good, shows each request's cost and saving on a dashboard, and can run optional side-by-side checks on a sample of your traffic. It does not provide provider failover. Setup is two settings, the base URL and your PromptUnit key, and your own provider key stays:

from openai import OpenAI

# Before: direct to OpenAI
client = OpenAI(api_key="sk-...")

# After: through PromptUnit (two settings: the base URL and your PromptUnit key)
client = OpenAI(
    api_key="sk-...",  # your own OpenAI key stays
    base_url="https://www.promptunit.ai/api/proxy/openai",
    default_headers={"x-promptunit-key": "YOUR_PROMPTUNIT_KEY"},
)

You do not change the calls themselves. Requests with tool results in the conversation stay on the model you asked for, and PromptUnit handles chat requests only (no embeddings, image generation or audio).

---

How PromptUnit routes, request by request

Routing changes which model answers your users, so PromptUnit makes the decision one request at a time:

• Every request is checked, from the first one, and classified across 10 signal dimensions.
• When a cheaper model is just as good for that request, PromptUnit uses it. When it isn't, the request goes to the model you asked for.
• Hard tasks and sensitive topics (medical, legal, money) get a stricter bar: they stay on the model you asked for or move only to one that is nearly as strong. Crisis topics never move.
• If you connected keys for more than one provider, the cheaper model can be at another of those providers.
• With the optional side-by-side checks on (off by default, paid on your own provider keys within a daily budget you set), a step of your app moves to a cheaper model on its own only after at least 20 judged checks with 9 in 10 as good and at least 10% lower cost. It moves back if fewer than 7 in 10 stay as good, and every move shows on the dashboard with an undo.

In the terms of this guide, that is a classifier with an evaluation loop on top. The dashboard shows each request with the model you asked for, the model that answered, what it cost and what it saved, and groups requests into the steps of your app.

To keep a request on the model you asked for, send the header x-promptunit-routing: off. For the cost side, see OpenAI API Cost Too High? Six Fixes, Ordered by Effort.

---

Common routing mistakes

Routing by model name in your code instead of in one place

# Fragile: routing logic hard-coded in application code
if task_type == "summary":
    model = "gpt-6-luna"
elif task_type == "analysis":
    model = "gpt-6-astra"

This scatters routing logic across the codebase, makes it hard to update, and is invisible to monitoring.

Routing without quality monitoring

You route 60% of traffic to a cheaper model and your costs drop. Three weeks later, customer satisfaction scores fall. Did routing cause it? Without per-model quality metrics, you can't tell.

Treating benchmark savings as your savings

The same RouteLLM routers saved over 85% on MT Bench and 35% on GSM8K. Your number depends on your mix of requests, so measure it on your own traffic.

Using a single model as ground truth for quality

"If the flagship wrote it, it is the gold standard" is not an evaluation method. Define task-specific quality metrics and measure both models against them, and if a model judges, say so and check its verdicts.

Routing high-stakes requests without human evaluation

Some tasks (medical, legal, financial, safety-critical) should not be routed on automated classifiers alone. Set explicit override rules for high-stakes categories.

---

Key takeaways

• LLM routing sends each request to the model best suited to it instead of one default model, balancing cost, quality and latency.
• Six common strategies: rules, classifiers, preference-trained routers, cascades, semantic routing, and latency or fallback routing. Production routers often combine several.
• The research reports large benchmark savings (RouteLLM: 35% to over 85% at 95% of GPT-4's performance; FrugalGPT: up to 98% on one dataset), and also the limits: routers trained on other data can be near random, and cascades depend on an accurate judge.
• Evaluate a router with a cost-quality curve against a simple baseline, on your own traffic, and check the judge.
• Cross-provider routing widens the choice of models. It adds resilience only if the router is built to fail over.
• Log every routing decision: which model, why, at what cost.
• Our own data is small and early: 18 of 24 comparisons matched at roughly 89% lower cost on those calls, judged by the model the app chose.

For the financial case against single-model defaults, see The Hidden Cost of Defaulting to Your Flagship Model. For routing decisions within one vendor, see Claude Haiku vs Sonnet: Routing Guide and GPT-5.4 vs GPT-5.4 Mini.

See the PromptUnit routing page for how the per-request check and the optional side-by-side checks work, connected with two settings: the base URL and your PromptUnit key.

---

Try it free

See where your AI budget goes. PromptUnit shows every request with the model that answered, what it cost and what it saved, starting with your first one. It is free until it has saved you money, then 20% of what it saves, and nothing if it saves nothing.

Start free, or talk to us if you want a walkthrough.