Groq for LLM Routing: Speed, Price and When to Use It
Groq serves OpenAI's open GPT OSS models at up to about 1,000 tokens per second, from $0.075 per million input tokens. Which workloads to move, with real math.
Groq sells speed. Its model page lists GPT OSS 20B at about 1,000 output tokens per second and GPT OSS 120B at about 500, so a 1,000-token answer streams in one to two seconds once it starts. The prices are low too: GPT OSS 20B costs $0.075 per million input tokens and $0.30 per million output tokens, half of GPT-4o-mini's $0.15 and $0.60. GPT-4o costs $2.50 and $10.00.
All prices in this post are list prices per million tokens from Groq's model page and OpenAI's pricing page, checked October 8, 2026.
Groq builds its own inference chips (it calls them LPUs) and runs open-weight models on them. It does not train models, and it does not compete on general-purpose compute. It competes on how fast tokens come out, and that is a routing dimension most teams ignore. Most production teams still send nearly all of their traffic to OpenAI or Anthropic. This post is about why, what it costs, and which workloads are the obvious candidates to move.
What Groq serves today
Groq's public lineup is small, and it changes. On August 16, 2026, Groq shut down Llama 3.1 8B and Llama 3.3 70B for its regular tiers; both are now enterprise-only with no published price, and Groq recommends GPT OSS 20B and GPT OSS 120B instead (deprecations page). The public text models today:
| Model on Groq | Input | Output | Listed speed |
|---|---|---|---|
| GPT OSS 20B | $0.075 | $0.30 | about 1,000 tokens/s |
| GPT OSS 120B | $0.15 | $0.60 | about 500 tokens/s |
| Qwen3.8 27B | $0.80 | $4.00 | about 450 tokens/s |
Two things stand out. GPT OSS 20B is half the price of GPT-4o-mini. GPT OSS 120B costs exactly what GPT-4o-mini costs, so the reason to choose it is speed and, on some tasks, quality, not price. And the churn itself is a lesson: a stack that hard-codes one provider's model names breaks when the provider retires them.
Speed is a cost lever
When a call returns in one second instead of several, that is not only a nicer interface. It cuts:
- Connection-holding time on your inference workers
- Memory pressure on streaming buffers
- Tail-latency risk during traffic spikes
- The waiting time that agent steps add up
If one user request fans out into 8 to 12 LLM calls that run one after another, the speed of each call decides whether the whole request takes seconds or minutes. For agent products that is a survival metric, not a nice-to-have. OpenAI and Anthropic do not list comparable speeds on their pricing pages, so measure your current provider on your own requests before you compare.
Let each request use the cheapest model that's just as good
PromptUnit checks each request your app sends. When a cheaper model is just as good for that request, it uses it; otherwise your model answers. Free until it has saved you money.
Try it free →The routing math on a real workload
Take a B2B SaaS company with four workloads, counted in tokens per month:
- A classifier ("which support category does this email fall into?"): 100M input, 2M output
- A summarizer ("summarize this thread for the agent"): 200M input, 30M output
- A Q&A endpoint ("answer this user question from our docs"): 50M input, 15M output
- A reasoning step ("plan the next action in this agent loop"): 20M input, 10M output
Default route, everything on GPT-4o ($2.50 / $10.00):
- Classifier: $250 + $20 = $270
- Summarizer: $500 + $300 = $800
- Q&A: $125 + $150 = $275
- Reasoning: $50 + $100 = $150
- Total: $1,495 a month
Routed by workload:
- Classifier on Groq GPT OSS 20B ($0.075 / $0.30): $7.50 + $0.60 = $8.10
- Summarizer on Groq GPT OSS 120B ($0.15 / $0.60): $30 + $18 = $48
- Q&A on GPT-4o-mini ($0.15 / $0.60): $7.50 + $9 = $16.50
- Reasoning on GPT-4o, where it earns its price: $50 + $100 = $150
- Total: $222.60 a month
That is about 85% lower on these assumptions, if the cheaper models hold quality on the first three workloads. That condition is the whole game, and you have to test it on your own requests. The reasoning step stays where it is because it is the one place the expensive model earns its price.
This is the same idea the LLM model routing guide walks through: the savings do not come from picking the cheapest provider. They come from sending each workload to the model that wins on its own mix of quality, cost and speed.
Why teams keep defaulting to OpenAI
If the math is this clean, why does the average production stack still send nearly everything to OpenAI?
Reason one: the integration cost is not zero. Groq's API is OpenAI-compatible, but compatible does not mean identical. Tool-calling details, streaming events and error formats can differ in ways that break code written against one provider's quirks. A team that ships an untested cross-provider call usually rolls it back within a week.
Reason two: model selection paralysis, now with churn. Picking the right model per workload needs evaluation infrastructure most teams do not have, and a provider's lineup can change within months, as Groq's did in August. The default is to do nothing.
Reason three: the tail risk on quality. A workload that is right 99.5% of the time on GPT-4o might be right 99.1% of the time on a smaller model. For a classifier, that is fine. For a customer-facing answer, the gap can end up in a screenshot. Without a check that catches regressions, teams stay on the safe route.
The broader version of this trap is in the hidden cost of defaulting to your flagship model. Groq is the speed-tier version of the same story: a cost and speed lever that most teams know exists and still do not pull.
Three workloads where Groq is worth testing first
If you do not want to overhaul your routing this quarter, three workload types can pay for the integration on their own:
1. Classifier and router calls. Anything where the output is a label, a score or a routing decision. GPT OSS 20B costs half of GPT-4o-mini per token and is fast enough that your classifier stops being the latency bottleneck. Test it on a few hundred of your own labeled examples first.
2. Streaming-first interfaces. Chat screens where a person watches the first token appear. Fast first tokens are the difference between "this app feels alive" and "this app feels broken."
3. Fan-out agent loops. Anything with many LLM calls per user request. Each call's speed adds up to the time the user waits. Routing the fan-out calls to a fast model while keeping the final synthesis on a stronger one is one of the cleanest two-tier patterns there is.
What about the quality risk?
It is real, and it can be bounded. Smaller open-weight models are not a drop-in for frontier models on tool-heavy agent work, long-context reasoning or subtle writing. They are often good enough for classification, extraction, summarization and structured output, but "often" is not a guarantee.
The way to bound the risk is not to avoid Groq. It is to run an evaluation on your own workload before you move any traffic, and to keep your previous route ready as a fallback in case quality shifts. That is the routing-layer pattern, not the swap-vendors pattern.
How PromptUnit fits
PromptUnit works with your own Groq key, as well as OpenAI, Anthropic, Google, DeepSeek and OpenRouter keys. It checks each request and uses a cheaper model only when it expects the answer to be just as good; otherwise the request stays on the model you asked for, and if a cheaper model's call fails, it retries on the model you asked for. The dashboard shows every request with the model that answered, what it cost and what it saved.
See also
Try it free
See exactly where your AI budget is going. PromptUnit shows every request with the model that answered, what it cost and what it saved, starting with your first one.
Start free, no credit card needed. Or talk to us if you want a walkthrough.