LLM Provider Failover Guide: Errors, Timeouts and Testing

Provider failover means your app can answer from a second provider when the first one fails. Whether it works depends on details: which errors to fail over on, how long to wait, and whether you tested any of it. This guide covers those, with Python code.

What has to keep working

Start with calls where a person is waiting. Such a call ends in an answer, an error, or silence until your timeout fires, and the third is easy to overlook. OpenAI's Python SDK waits up to 10 minutes by default and retries some failures twice on its own (README). Anthropic's Python SDK has the same two defaults (docs). Work that can wait, like nightly jobs, can sit in a queue and retry later; the batch API post covers which work qualifies.

Pick a failover pattern

Manual

Someone changes the key or base URL and redeploys. Time it in a drill instead of guessing.

Primary with a standby

Traffic goes to the primary. After repeated failures a circuit breaker opens and traffic goes to the standby, and after a wait one trial call tests the primary. This is active-passive, and the code below does it.

Both live

Traffic splits across providers all the time (active-active), so you must accept either provider's answer on any request. AWS's Route 53 docs explain both terms.

Libraries provide the breaker: Resilience4j (Java), Polly (.NET) and pybreaker (Python). Tenacity retries calls but has no breaker. Check defaults: Resilience4j waits for 100 recorded calls before it calculates a failure rate, so lower that on low traffic. pybreaker opens after 5 consecutive failures.

Check how the standby speaks

Anthropic's Messages API differs from OpenAI's Chat Completions in three ways: the system prompt is a top-level system field with no system role, max_tokens is required, and the reply content is a list of blocks (docs).

Many providers also accept OpenAI-format requests, so a standby client may need only a new base URL, key and model name. Their docs list the limits:

Provider	OpenAI-format base URL	Documented limits
Anthropic	https://api.anthropic.com/v1/	Meant for testing, not production for most uses. Ignores response_format and strict. No prompt caching. System messages merge into one.
Google	https://generativelanguage.googleapis.com/v1beta/openai/	OpenAI library support is in beta.
Groq	https://api.groq.com/openai/v1	400 error for logprobs, logit_bias and top_logprobs.
DeepSeek	https://api.deepseek.com	Also takes Anthropic format at /anthropic.

Anthropic's quiet ignoring is the risky kind: JSON mode can stop being enforced with no error. Test every parameter you rely on against the standby, and keep a model name per provider in config. State kept by one provider, such as previous_response_id, does not exist on another (Groq lists it as unsupported), so keep your own message history. The standby also receives your users' prompts, so read its data terms.

To skip building this layer, gateways document provider fallback: OpenRouter, LiteLLM and Portkey. Our OpenRouter, LiteLLM and PromptUnit comparison and PromptUnit vs Portkey show how they differ.

Know which errors to fail over on

The table groups what the official docs describe. The last column is our suggested action.

Signal	Where the docs list it	Suggested action
429 rate limit	OpenAI and Anthropic (may carry retry-after), Gemini, Groq	Wait the stated time if it is short, else fail over
429, 402 or 400 for credit, spend cap or daily quota	OpenAI (credit_balance_exhausted, spend and usage limit codes), Anthropic (tier spend cap, billing_error, 400 for limits you set), Gemini, DeepSeek	Do not retry. Fail over and alert the account owner
500, plus 502, 503 or 504	All five providers	Retry once after a short random pause, then fail over
529	Anthropic overloaded_error	Treat as 5xx
498	Groq, flex tier at capacity	Fail over
Timeout or dropped connection	SDK APITimeoutError, APIConnectionError	Fail over, since a retry costs another full timeout
400, 401, 403, 404, 413, 422	Client errors in every provider's docs	Do not fail over. Fix the request, key or region and alert someone

A 429 can mean slow down or out of money. OpenAI's docs say billing, spend and quota errors are fixed by adding credits or raising limits, so retrying them only adds delay (error codes). Anthropic's spend-cap 429 has no retry-after header (errors). DeepSeek's docs suggest switching providers for a while when you hit its rate limit (errors). Gemini and Groq list their codes too (Gemini, Groq).

A sample you can adapt

This version fails over on timeouts, 5xx errors, rate limits it cannot wait out, and exhausted credit. It uses OpenAI's Python SDK and pybreaker.

import random
import time

import pybreaker
from openai import (
    APIConnectionError, APIStatusError, APITimeoutError,
    InternalServerError, OpenAI, RateLimitError,
)

# PRIMARY_MODEL, STANDBY_MODEL, STANDBY_API_KEY, STANDBY_BASE_URL: your config.
TIMEOUT = 20.0  # seconds; set from your own latency data (SDK default: 10 minutes)
MAX_WAIT = 5.0  # longest retry-after we will sit through

primary = OpenAI(timeout=TIMEOUT, max_retries=0)  # reads OPENAI_API_KEY
standby = OpenAI(api_key=STANDBY_API_KEY, base_url=STANDBY_BASE_URL,
                 timeout=TIMEOUT, max_retries=2)

# Client errors are our mistakes, not an outage, so they must not trip the breaker.
breaker = pybreaker.CircuitBreaker(
    fail_max=5,
    reset_timeout=30,
    exclude=[lambda e: isinstance(e, APIStatusError)
             and e.status_code < 500 and e.status_code not in (408, 429)],
)

OUT_OF_MONEY = {"credit_balance_exhausted", "organization_spend_limit_exceeded",
                "project_spend_limit_exceeded", "organization_usage_limit_exceeded"}


def out_of_money(err):
    return err.type == "insufficient_quota" or err.code in OUT_OF_MONEY


def retry_after(err):
    try:
        return float(err.response.headers["retry-after"])
    except (KeyError, ValueError):
        return None


def call_primary(messages):
    for attempt in (1, 2):
        try:
            return primary.chat.completions.create(model=PRIMARY_MODEL, messages=messages)
        except RateLimitError as err:
            wait = retry_after(err)
            if attempt == 2 or out_of_money(err) or wait is None or wait > MAX_WAIT:
                raise
            time.sleep(wait)
        except APITimeoutError:
            raise  # a retry would cost another full timeout
        except (APIConnectionError, InternalServerError):
            if attempt == 2:
                raise
            time.sleep(random.uniform(0.2, 1.0))


def ask(messages):
    try:
        return breaker.call(call_primary, messages)
    except (pybreaker.CircuitBreakerError, RateLimitError,
            APIConnectionError, InternalServerError):
        return standby.chat.completions.create(model=STANDBY_MODEL, messages=messages)

• max_retries=0 stops the SDK's own retries (by default it retries connection errors, 408, 409, 429 and 5xx twice), so your code decides.
• The standby keeps the SDK's two retries because nothing sits behind it.
• While the circuit is open, pybreaker raises CircuitBreakerError. After reset_timeout it lets one call through to test the primary.
• exclude accepts a function as well as exception classes, as the pybreaker README documents. Upgrade pybreaker if your version rejects one.

Keep retries few and spaced out. OpenAI's write-up of its April 20, 2026 incident (approximately 7:18 AM to 9:00 AM PDT; ChatGPT availability for some users, plus login and signup errors on platform.openai.com and the Codex app) says retry, caching and routing behavior in its systems amplified an overload, and lists better circuit breakers and retry behavior among its fixes (write-up). Our recap covers what it means for apps.

Streams can fail after a 200

A streaming call can fail after the HTTP status is already 200. Anthropic documents errors that arrive as events inside the stream, such as overloaded_error, the streaming counterpart of a 529 (docs). OpenAI's SDK raises APITimeoutError or APIConnectionError while you read a stream and does not retry it, because a replay could repeat output you already delivered. Decide ahead of time what happens once text has reached the user: restart on the standby, or show an error.

Health checks and alerts

Your own traffic is the best signal: error rate and latency per provider and per feature. The observability metrics guide covers which error and retry rates to track. Add a tiny real request on a schedule so you get a signal when traffic is low.

OpenAI and Anthropic status pages both serve JSON at /api/v2/status.json (OpenAI, Anthropic). They show what the provider has posted, not what your calls see right now.

Use three alert levels: a probe or status page shows trouble; the breaker opened; the standby is failing too.

Test the failure, not the happy path

Point the primary client's base URL at a stub you control that can return a 429 with retry-after, a 429 for exhausted credit, a 500, a 503, a 529, a response that never arrives, and a stream that stops after the first chunk. Then check that the breaker opens after five failures and the standby answers, that one trial call reaches the primary after the reset time, that a 400 or 401 reaches your logs without tripping the breaker, and that your logs say which provider answered.

On Kubernetes, Chaos Mesh NetworkChaos has a partition action, and externalTargets takes domains with direction: to. Its HTTPChaos does not inject into HTTPS unless you set up its TLS mode.

Finally, measure your own switch time, from the first failed call to the first good answer from the standby, and put that number in your runbook.

What a proxy covers, and what it does not

PromptUnit is a proxy you point your app's AI API base URL at. It checks each request and uses a cheaper model when that model is just as good, and otherwise sends the request to the model you asked for. If a cheaper model it chose fails, PromptUnit retries on the model you asked for. That is model fallback.

It does not promise failover when a provider is down. If the provider behind the model you asked for fails, PromptUnit does not reroute the request to another provider, and your request can fail. Any proxy is also one more hop, so keep a way to point the base URL straight at the provider without a deploy.