LLM Model Routing: The Complete Guide
A complete guide to LLM model routing: how it works, strategies that cut costs 40-70%, quality validation, and implementing it without a codebase rewrite.
LLM model routing is the practice of automatically directing each API call to the model best suited for that specific request, balancing cost, quality, and latency, rather than sending all requests to a single default model.
Done well, routing cuts LLM inference costs by 40–70% without degrading user-facing quality. Done poorly, it introduces quality regressions that take weeks to diagnose. This guide covers how to do it well.
What routing looks like on real traffic
PromptUnit runs inside live products, so we can measure this instead of estimating it. For a sample of real requests we run a cheaper model on the same prompt and have an independent model judge it against the model the app chose. What that shows so far:
- A cheaper model matched or beat the chosen model about 75% of the time.
- On those calls, the cheaper model cost roughly 89% less per call.
- The best swap depends on the pair: Claude Sonnet 5 to Gemini 2.5 Flash matched the chosen model in every test we ran, while drops to the very smallest models held quality less often.
We report these as shares, not a fixed count, because the dataset grows as more traffic runs through, and we store only metadata (task type, model, cost, judge verdict), never prompt text. The takeaway for routing: the opportunity is large, but it has to be checked per task and per model pair, which is what the rest of this guide covers.
Why Routing Exists
The LLM market has fragmented into a spectrum of model tiers:
| Model Tier | Example Models | Relative Cost | Relative Capability |
|---|---|---|---|
| Frontier | GPT-5.4, Claude Opus 5.5, Gemini 3.1 Pro | $$$$ | Highest |
| Advanced | GPT-5, Claude Sonnet 5, Gemini 3.5 Flash | $$$ | High |
| Mid-tier | GPT-5 mini, GPT-4o-mini, Claude Haiku 4.5 | $$ | Good |
| Efficient | GPT-5 nano, Gemini 3.1 Flash-Lite, DeepSeek Flash | $ | Task-specific |
The price difference between frontier and efficient models is 20x–100x. The capability difference on most real-world tasks is much smaller.
A team that sends every API call to a frontier model is paying frontier prices for mid-tier tasks. Routing exists to match the model to the task, pay frontier prices only for tasks that need frontier capability.
Router, gateway, proxy: the terms, defined
These words get used interchangeably, so here is the plain version.
An AI router decides, per request, which model should answer. It reads the request, judges how hard it is, and sends it to the cheapest model that can handle it well. Routing is the decision.
An LLM gateway is the single entry point your app sends every AI call through. It forwards the call to the right provider and adds the things you would otherwise build yourself: routing, cost tracking, rate limiting, caching, and failover. The gateway is the layer; routing is one feature inside it.
An AI inference proxy is how a gateway usually works in practice. Your app points its base URL at the proxy instead of at OpenAI, the proxy forwards the request with your own provider key, and it adds routing and tracking on the way through. Nothing else in your code changes. The proxy is the plumbing.
In short: the proxy is where the call travels, the gateway is the set of features it passes through, and the router is the decision that picks the model. PromptUnit is all three: a proxy you point your base URL at, a gateway that tracks cost and enforces limits, and a router that switches to a cheaper model when it is just as good.
The Three Core Routing Strategies
1. Static rule-based routing
The simplest approach: you write explicit routing rules based on feature, endpoint, or request metadata.
def route_request(feature: str, messages: list) -> str:
# Static routing by feature
if feature in ["code_review", "architecture_analysis"]:
return "gpt-4o"
elif feature in ["summarization", "classification", "extraction"]:
return "gpt-4o-mini"
else:
return "gpt-4o" # default
Advantages: Predictable, auditable, no latency overhead.
Disadvantages: Brittle. Feature tags don't always reflect actual complexity. A "summarization" feature might sometimes receive complex multi-document requests. New features start unrouted by default. Rules accumulate and become unmaintainable.
Static rules are a reasonable starting point for small teams with well-understood traffic. They break down as the application grows.
2. Content-based classification routing
The proxy (or a classifier in your stack) reads the actual request content and makes a routing decision based on complexity signals:
- Token count of the input
- Presence of code, structured data, or domain-specific terminology
- Instruction complexity (single vs. multi-step)
- Context window depth (multi-turn conversation length)
- Explicit complexity markers in the prompt
def classify_complexity(messages: list) -> str:
total_tokens = estimate_tokens(messages)
has_code = any("```" in m["content"] for m in messages if "content" in m)
is_multistep = count_instructions(messages) > 3
is_long_context = total_tokens > 8000
if has_code and is_multistep:
return "gpt-4o"
elif is_long_context:
return "gpt-4o"
elif total_tokens < 2000 and not is_multistep:
return "gpt-4o-mini"
else:
return "gpt-4o" # safe default for ambiguous cases
Advantages: Adapts to actual request content, not just feature labels. Works for heterogeneous traffic on the same endpoint.
Disadvantages: Classifier accuracy matters, wrong classifications on edge cases degrade quality. Requires maintenance as task distribution changes.
3. ML-based adaptive routing
The most sophisticated approach: a learned model that predicts optimal routing based on historical request-response pairs and quality signals.
The system trains on your specific traffic:
- Input: request features (token count, task type signals, prompt complexity)
- Label: which model produced acceptable quality at minimum cost?
- Output: routing probability distribution across available models
Advantages: Learns from your actual traffic, improves over time, handles edge cases that rule-based systems miss.
Disadvantages: Requires significant traffic volume to train on, latency for model inference, cold-start problem for new traffic patterns.
This is the approach used by purpose-built routing infrastructure. It's not practical to implement as a custom build for most teams, the engineering overhead exceeds the value unless you're at significant scale.
Quality Validation: The Part Most Guides Skip
Routing is only valuable if quality is preserved. This is the hard part.
Defining quality
Quality means different things for different tasks:
- Classification: Accuracy rate (verifiable, has ground truth)
- Summarization: Human preference scores, factual accuracy, key point coverage
- Code generation: Test pass rate, syntactic correctness, reviewer acceptance
- Customer support: Resolution rate, customer satisfaction scores, escalation rate
- Creative generation: Human preference (subjective, difficult to automate)
Tasks with ground truth (classification, extraction, structured output) are easy to validate at scale. Tasks with subjective quality (creative writing, nuanced support) require human evaluation, which doesn't scale.
The shadow testing approach
Before routing live traffic, run both models in parallel on a sample:
async def shadow_test(messages: list, sample_rate: float = 0.05):
"""Run a small fraction of traffic through both models for comparison."""
import random
if random.random() > sample_rate:
return None # Not a shadow test request
# Run both models concurrently
results = await asyncio.gather(
call_model("gpt-4o", messages),
call_model("gpt-4o-mini", messages),
)
log_comparison(
messages=messages,
frontier_response=results[0],
cheaper_response=results[1],
timestamp=datetime.utcnow(),
)
return results # Return both for evaluation
Shadow test results populate a comparison dataset you can evaluate against your quality rubric. This tells you, with data, whether routing is safe for your specific traffic, before it affects any user.
Ongoing quality monitoring
Routing changes the model distribution. As it does, you need to monitor for quality regressions:
- Regression signals to watch: User corrections, retry rates, explicit negative feedback, task-level downstream metrics (conversion rate on content generated by each model)
- Drift detection: If your traffic patterns shift (new features, new user segments), previously-safe routing rules may need revisiting
- Per-model quality scores: Maintain rolling quality metrics by model for each task category
The Routing Decision Architecture
For production systems, the routing decision needs to be fast, reliable, and auditable.
Incoming Request
↓
Feature Extraction
- Token count
- Task type signals
- Code/data detection
- Multi-step instruction count
- Context depth
↓
Routing Classifier
- Rule-based threshold checks
- ML-based complexity score (if applicable)
- Override rules (always gpt-4o for feature X)
↓
Model Selection
- Primary model
- Fallback model (if primary is rate-limited or down)
↓
Request Forwarding
↓
Response + Routing Metadata
- Which model was used
- Why it was selected
- Cost of this call
- Quality confidence
Every routing decision should be logged with enough information to audit it later. "Why did this request go to gpt-4o-mini?" should always be answerable from the logs.
Cross-provider routing: the next step up
Single-model routing (a flagship vs its own mini) is the most common starting point. Cross-provider routing extends the decision across vendors, because the best model for a task is not always the cheapest one from the vendor you happen to use:
| Use Case | Candidate Models | Why |
|---|---|---|
| Complex reasoning | GPT-5.4, Claude Opus 5.5, Gemini 3.1 Pro | Compete on quality |
| Cost-optimized routine work | GPT-4o-mini, Gemini 3.1 Flash-Lite, Claude Haiku 4.5 | Compete on price |
| Code generation | Claude Sonnet 5, GPT-5.4, DeepSeek V4 Pro | Compete on code quality |
| Long context | Gemini 3.5 Flash (1M tokens), Claude Sonnet 5 | Compete on context length |
Different vendors price different tiers differently, and the strongest model on a given task is not always the most expensive. Routing per task across vendors exploits both facts at once.
It is measurable, not theoretical. PromptUnit samples real requests, runs a cheaper model from another vendor on the same prompt, and has an independent model judge the result against the model the app chose (metadata only, never prompt text). The clearest result so far is cross-vendor: Claude Sonnet 5, answered instead by Google's Gemini 2.5 Flash, matched the original in every test we ran. Across all pairs, a cheaper model held the quality about 75% of the time, at roughly 89% less cost on those calls.
Cross-provider routing also adds resilience: if one vendor has an outage or a rate-limit surge, the router can send the request to another. And it is insurance against pricing changes, when one vendor raises prices, routing shifts traffic to the others without a code change.
The catch is integration: every provider has a different API, and a cheaper vendor only helps if its answer is actually as good, so each pair has to be quality-checked before it is trusted. That is exactly why an inference proxy is the right tool for this. Your application sends one format to one endpoint; the proxy speaks every provider's API and runs the quality checks for you.
Implementing Routing Without Custom Engineering
Building routing from scratch involves:
- A classifier (rule-based or ML-based)
- A model registry (which models are available, at what cost, with what rate limits)
- A routing decision engine
- Fallback and retry logic for each provider
- Quality monitoring and alerting
- A logging and cost attribution layer
- A configuration system so rules can be updated without code deploys
This is 4–8 weeks of engineering for a senior engineer, with ongoing maintenance overhead. It's defensible at a company with significant LLM scale and an infrastructure team. It's hard to justify for most product teams.
The alternative is an inference proxy that provides all of this at the infrastructure layer. PromptUnit implements full routing, quality monitoring, cost attribution, and provider failover, activated by pointing your SDK at a different base URL:
# Before: direct to OpenAI
client = OpenAI(api_key="sk-...")
# After: routed through PromptUnit
client = OpenAI(
api_key="sk-...",
base_url="https://www.promptunit.ai/api/proxy/openai",
default_headers={"x-promptunit-key": "YOUR_KEY"},
)
Everything else stays the same. Existing tests, streaming, function calling, and error handling all continue to work.
Routing From the First Request
Routing changes which model handles each request. That's a meaningful change to a production system, so PromptUnit makes it one request at a time.
Savings start with the first request:
- Every request is checked
- When a cheaper model is just as good for that request, PromptUnit uses it
- When it isn't, the request goes to the model you asked for
- A quality setting controls how cautious the switching is
The dashboard shows each request within seconds: the model you asked for, the model that answered, what it cost and what it saved.
To keep a request on the model you asked for, send the header x-promptunit-routing: off. Read more about how this works in practice in How to Reduce Your OpenAI API Costs by 50–70% Without Changing Your Code.
Common Routing Mistakes
Routing by model name in your code instead of using a proxy
# Fragile: hard-coded routing logic in application code
if task_type == "summary":
model = "gpt-4o-mini"
elif task_type == "analysis":
model = "gpt-4o"
This scatters routing logic across the codebase, makes it hard to update, and is invisible to monitoring.
Routing without quality monitoring
You route 60% of traffic to a cheaper model and your costs drop. Three weeks later, customer satisfaction scores fall. Did routing cause it? Without per-model quality metrics, you can't tell.
Using a single model string as ground truth for quality
"If it was good enough for GPT-4o, it's the gold standard" is not a quality evaluation methodology. Define task-specific quality metrics and measure both models against them.
Routing high-stakes requests without human evaluation
Some tasks, medical, legal, financial, safety-critical, should not be routed based on automated classifiers alone. Establish explicit override rules for high-stakes categories.
Key Takeaways
- LLM model routing directs API calls to the optimal model per request, based on complexity, cost, and task type, instead of defaulting everything to the most capable (and most expensive) model.
- Three routing strategies exist: static rules (simple but brittle), content-based classification (adapts to actual content), and ML-based adaptive routing (learns from your traffic). Most teams start with rules and evolve to classification.
- Quality validation is the essential step most routing implementations skip. Shadow testing, quality rubrics, and ongoing monitoring are non-negotiable for production systems.
- Multi-provider routing extends savings and adds provider resilience, but requires an abstraction layer that speaks multiple provider APIs.
- Building routing from scratch is 4–8 weeks of engineering with ongoing maintenance. Purpose-built inference proxies provide this at the infrastructure layer for a fraction of the engineering cost.
- The right way to deploy routing is one request at a time: use a cheaper model only when it is just as good for that request, and keep the rest on the model you asked for.
- Every routing decision should be auditable: log which model was chosen, why, at what cost, and with what quality confidence.
For teams spending over $1,000/month on LLM APIs, intelligent routing is the highest-ROI infrastructure investment available. The engineering overhead is low; the financial return is immediate. Read about what actually happens when you analyze your production traffic patterns.
For the financial case against single-model defaults, see The Hidden Cost of Defaulting to GPT-4o.
See our LLM Routing solution for how PromptUnit implements the classification, quality threshold, and adaptive-learning pieces of this guide as production infrastructure, activated by a one-line base URL change.
Try It Free
See exactly where your AI budget is going. PromptUnit shows every request with the model that answered, what it cost and what it saved, starting with your first one.
Start free, no credit card needed. Or talk to us if you want a walkthrough.