How your savings grow over time

Each request is checked on its own. It goes to the cheapest model, among the providers you connected, that scores within a few points of the model you asked for on that kind of task and clears your quality bar. If none does, it stays on your model.

Some requests always stay on the model you asked for: requests with tool calls, judging steps such as reviews, grading and fact checks, questions about current events, and crisis topics. Requests with images stay too, unless the side-by-side checks (section 4) prove a cheaper model on them. Medical, legal and money topics get a stricter bar.

A request only moves to another provider's model if you connected that provider's key too. With one key, it moves between that provider's own models.

Accounts created from our homepage start with the quality setting at 75, so savings show early. Other accounts start at 85, the careful setting, so the check at the end of the window changes nothing for them.

Answers are scored automatically: not empty, not a refusal, not cut off, in the format asked for, a sensible length. It checks the form of the answer, not its facts. Answers your app receives as a stream are scored the same way once the stream ends; answers that are tool calls are not scored. When your app sends the same request again within 90 seconds, that counts as a repeat, but only for requests a cheaper model answered: sending again a request that stayed on your model says nothing about cheaper models.

The dashboard shows "Learning: 7 of 20 requests" while this runs, then the result. If the cheaper models' answers scored poorly (an average under 60), or your app sent more than one in four of them again (once at least 5 requests went to a cheaper model), the quality setting goes up to 85 that night.

Every night PromptUnit looks at your last 7 days, for each step of your app and each kind of request: classification, summaries, extraction, and so on.

• A step of your app with at least 5 scored requests that week gets its own bar, from its own results, instead of the bar for its kind of request. Two steps of the same kind can then be treated differently, and the steps list on your dashboard shows each step's change.
• Answers that score well and are rarely repeated: the bar for that step or kind of request drops 2 to 3 points, so cheaper models qualify more often.
• Answers that score poorly or are often repeated: the bar rises 5 to 10 points, back toward stronger models.
• When the results go back to normal, so does the bar.
• It needs at least 5 scored requests of that kind in the week. With fewer, nothing changes.
• Repeats count once at least 5 requests of that kind went to a cheaper model in the week. Until then the bar can rise on poor scores, but it doesn't drop.

It also corrects each model's score by how it actually did on your traffic over the last 30 days, once a model has at least 10 scored answers of a kind: up to 10 points higher or 25 points lower. A cheaper model that does well on your requests gets picked more, and one that does badly stops being picked.

You can see this under Settings, in Routing Health: the quality, the repeats, the number of requests and the current adjustment for each kind of request.

On a sample of your real requests, after the answer has already gone back to your app:

• If PromptUnit moved a request to a cheaper model, the model you asked for answers it too and judges whether the cheaper answer was as good. A step of your app where the cheaper answers come out worse stops being moved.
• Up to two cheaper models from other providers answer the same request and are judged the same way, and so does one cheaper model from your own provider that PromptUnit doesn't use for that request yet. That is how a new model, like Claude Haiku 5.5, earns its place on your steps. On steps where PromptUnit tries your provider's cheaper models instead (section 5), your provider's models are tested here only to confirm a try.
• A model that came out worse at least 3 times on a step, and under 9 in 10, isn't tested there again for 30 days, so no money goes to checks it can't pass.
• Requests with images are checked too, on cheaper models that can read images, and your model sees the same images when it judges. Up to 4 images per request, and never on steps that check or grade work. A check with images costs more than one without, because the images go again to the cheaper model and to the judge.

These checks are off by default. They run on your own keys, so you pay for them directly: your OpenRouter key if you connect one (it lets them test other providers' models too), otherwise your Anthropic or OpenAI key. Never in Privacy mode, and they keep only scores, costs and timings, never your prompts or answers.

A check pays for extra answers: the cheaper model's, your model judging it, and for a request PromptUnit already moved, your model's own answer too. So one check can cost more than the request itself. That is why they have a daily budget you set: when it's used up, the checks stop until the next day, and while your steps are still being tested, it is usually used up. Setup offers to turn them on, at up to $1 a day, once your first request arrives; Settings on your dashboard changes that any time.

Steps move on their own. When a cheaper model keeps up on a step of your app (at least 20 checks, 9 in 10 judged as good as your model, and at least 10% cheaper), that step moves to it the night after: on your provider key for your provider's own models, on your OpenRouter key for another provider's. The checks keep running on the moved requests, and if fewer than 7 in 10 come out as good (at least 3 checks in 14 days), the step moves back on its own. Every move shows on your dashboard with an undo, and a model you undo is never used for that step again.

A step's requests with images move only when at least 20 checks with images came out 9 in 10 as good. Until then, if the step also sends requests without images, only those move, and the requests with images keep being checked on the side until they prove it too.

Under the status, your dashboard lists what could save you more, from your own last 7 days. Each item shows only when it applies to your app:

• A lower quality setting. For each request that stayed on your model because no cheaper model cleared the bar, PromptUnit also works out which model a setting 5 points lower would have chosen, with everything else about the request the same. Only that model's name is stored. The dashboard adds them up: how many requests would have moved, and what that would have saved. It is not suggested while the nightly adjustments are raising the bar somewhere, because then quality comes first.
• Requests that keep their model: ones with images or files, tools, or a reasoning mode such as extended thinking, with their share of your requests and what they cost. With the side-by-side checks on, requests with images can still move once the checks prove a cheaper model on them (section 4). If one of those steps is simple, you can also ask for a cheaper model there in your code.
• A second provider. With one provider's key, a request moves only between that provider's own models. When many requests stay put because none of them is good enough, another provider's key gives PromptUnit more models to choose from.
• Side-by-side checks: the steps where a cheaper model is on its way to taking over (how many checks it matched so far), or a pointer to turn the checks on.

• The quality setting, on the Routing page of your dashboard. Higher is more careful: fewer switches, smaller savings. Lower saves more.
• One request: send the header x-promptunit-routing: off and it stays on its model.
• All requests: the on/off switch on the Routing page.
• "Find bigger savings", in Settings: the side-by-side checks and the tries of cheaper models, with the daily budget for the checks.
• Privacy mode, in Settings, keeps only what billing needs (the models, tokens, costs and response times), and turns off the step names, the side-by-side checks and the tries. Without scores, the nightly adjustments have nothing to learn from.

Your prompts and answers. The scores, repeats and adjustments on this page come from metadata: which model answered, how many tokens, what it cost, and the automatic score of the answer's form.