LLM Observability: The Metrics That Actually Drive Decisions

Most teams track monthly LLM spend. That number tells you almost nothing about what to do next. You know you spent $8,400 last month. You don't know whether one poorly designed feature burned $3,000 of that, whether your cache hit rate dropped because someone put a timestamp in a system prompt, whether hidden reasoning tokens doubled your output bill, or whether retries have been quietly adding 15% to every invoice for two months. The monthly total is a budget metric. It is not an optimization metric.

Production LLM systems need observability built around the decisions you have to make: which feature to optimize, which model to replace, where latency hurts users, and where you pay for tokens nobody sees. Prices and API field names below come from OpenAI's and Anthropic's own documentation, checked on October 24, 2026.

The Metrics That Lead to Action

Cost per call by feature

This is the most important metric for cost optimization. Not total spend, and not the average cost per call across your whole system, but the cost of each call broken down by the product feature that triggered it. One feature often dominates: an onboarding flow that runs a large model with a long prompt, a report generator that passes full documents into context, or a search ranking function that sends every request to GPT-6 Astra ($10 per million input tokens and $50 per million output tokens for prompts up to 272K tokens) when GPT-6 Luna ($0.10 and $0.50) might do the job at one hundredth of the price. Whether it can is something you test on your own traffic, not something you assume. Without per-feature attribution you cannot find these features at all. LLM cost attribution by feature covers which fields each provider lets you group by.

Cost per successful task

Cost per call hides failures. A task is the unit your user cares about: one answered question, one processed document, one finished agent run. It can take several calls: retries, a re-ask after the output fails validation, the tool-use steps of an agent. Cost per successful task is the total cost of every call made for a set of tasks, failed attempts included, divided by the number of tasks that succeeded.

This is the metric that keeps a model swap honest. A cheaper model that needs a second attempt on a third of tasks, or more agent steps to finish, can cost more per success than the model it replaced, even though its cost per call is lower. Failed calls are not free either. OpenAI's reasoning guide notes that a response which hits max_output_tokens comes back with status incomplete, possibly before any visible output, and you are still charged for the input and reasoning tokens. To measure this metric, give every call a task ID and record whether the task succeeded.

Cost per output unit

Cost per output unit converts LLM spend into business terms. The right unit depends on your product: cost per document processed, per report generated, per user question answered. This is your AI unit economics. If your cost per document processed is $0.35 and you charge $0.10 per document, you have a margin problem whatever your monthly total is. If it's $0.04, you have pricing power. Monthly spend tells you neither.

Token efficiency ratio

This measures how much useful work you get out of the tokens you send. A simple version: divide useful output tokens by total tokens in the call (input plus output). If you send a 4,000-token context to extract a 30-token structured field, your ratio is poor, and the fix is probably prompt restructuring or retrieval redesign rather than a model switch. Teams with good ratios have usually trimmed system prompts, removed redundant few-shot examples and pruned their context. Teams that haven't done that work often find that prompt optimization saves more than any model change.

Cache hit rate

Cache hit rate is the share of input tokens served from the prompt cache: cached input tokens divided by total input tokens. It tells you what caching actually saves, as opposed to what it could save. Both providers report the counts in the usage object of every response, but the fields differ:

• OpenAI (Responses API): usage.input_tokens_details.cached_tokens for cache reads and usage.input_tokens_details.cache_write_tokens for cache writes. usage.input_tokens already includes both, so the hit rate is cached_tokens / input_tokens. OpenAI's prompt caching guide recommends this ratio, summed by user, workspace or day.
• Anthropic (Messages API): cache_read_input_tokens for reads and cache_creation_input_tokens for writes. Here input_tokens counts only the tokens after the last cache breakpoint, so total input is input_tokens + cache_creation_input_tokens + cache_read_input_tokens, and the hit rate is cache_read_input_tokens divided by that sum. Apply OpenAI's formula to Anthropic data and the rate can come out above 100%.

The money at stake is large. On Claude Sonnet 5.5 a cache read costs $0.10 per million tokens against $2 for uncached input. On GPT-6.1 Sol it is $0.10 against $2.00, and on GPT-6 Astra $1.00 against $10. Writes cost extra, though. A 5-minute cache write on Claude costs 1.25 times the base input price and a 1-hour write costs 2 times. OpenAI turns caching on by default for supported models, and on GPT-5.6 and later, including the GPT-6 models, a cache write costs 1.25 times the uncached input price. A prefix that is written and never read again costs 25% more than sending it uncached, so track write tokens next to the hit rate. When reuse works, the math is strongly in your favor: OpenAI's own example puts one write plus nine full reads at 2.15 times the prefix's ordinary input cost, against 10 times without caching.

If your hit rate is 40% when you expected 80%, something near the start of the prompt is changing between calls: a timestamp or user-specific value in what you thought was a stable system prompt, a reordered tool list, or a changed effort setting. Both providers list the reasoning or effort setting among the things that can break a cached prefix. Minimum lengths matter too. Anthropic does not cache prompts shorter than 512 tokens on Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, and returns no error when that happens. OpenAI's minimum is 1,024 tokens on GPT-5.6 and later. The prompt caching guide for OpenAI and Anthropic covers the setup on each side.

Reasoning token share

Current models think before they answer, and you pay for it. Reasoning tokens (OpenAI) and thinking tokens (Anthropic) are billed as output tokens even though you never see them in full. OpenAI does not return reasoning tokens through the API. On Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, thinking is on by default and the thinking text is omitted from the response unless you ask for a summary. OpenAI says its models can generate anywhere from a few hundred to tens of thousands of reasoning tokens depending on the problem, and at $50 per million output tokens on GPT-6 Astra or $20 on Claude Opus 5.5, that adds up.

Track reasoning tokens as a share of output tokens, per feature and per model. OpenAI reports the count in usage.output_tokens_details.reasoning_tokens and Anthropic in usage.output_tokens_details.thinking_tokens. In both APIs the count is already inside output_tokens, so do not add it a second time. Anthropic notes that its count comes from re-tokenizing the raw reasoning text, so it can be off by a few tokens.

A high share on a simple feature such as classification, extraction or short answers points to the effort setting. OpenAI's reasoning.effort defaults to medium on GPT-6.1 Sol, GPT-6 Sol and GPT-6 Luna. Anthropic's output_config.effort defaults to high on most Claude models, including Sonnet 5.5, and to medium on Opus 5.5 and Haiku 5.5. Lower effort uses fewer tokens on both platforms, at some cost in capability, so check your quality signal before and after you change it.

Quality score by model

This is what makes a model decision defensible. If you have any quality signal at all, whether human labels, an LLM-as-judge score, or a downstream proxy like conversion rate, user correction rate or the share of outputs that pass validation, track it per model and per feature next to cost. The combination tells you whether a cheaper model is still cheaper once quality is counted. A model that costs 60% less but produces output that needs human correction 30% of the time may cost more once you count the remediation. Record the model that actually answered and the effort setting it ran at, because effort changes both the cost and the output. This metric is the foundation of LLM model routing decisions that hold up in production.

Retry rate

Retry rate is one of the most undertracked metrics in production LLM systems. When a call fails on a timeout, a server error, a rate limit or an output that fails validation, your system retries, and each retry is a cost you didn't budget. With retries about as large as first attempts, a 10% retry rate makes your effective cost roughly 10% higher than your raw token cost, and a 25% retry rate means you pay for a quarter more calls than your usage suggests. Track it per feature and per model, separately from the raw error rate, because a call that succeeds on retry still shows up as a success in your quality metrics. Cost per successful task is where its price becomes visible.

Latency by percentile

Latency is a production health metric more than a cost metric, but it belongs in any complete observability stack. The p50 tells you the median experience. The p95 and p99 tell you what your heaviest users and hardest queries hit. Average latency almost always looks acceptable because it smooths over the long tail. A p99 of 45 seconds on a synchronous user-facing call is a product problem. A p99 of 45 seconds on a batch pipeline is fine. For streamed responses, also track time to first token, since that is what a user waits through before anything appears.

Metrics That Feel Useful But Aren't

Total monthly spend is a budget metric. Use it to set limits, flag anomalies and report to finance. Do not use it to decide what to optimize. By itself it tells you nothing about where to look.

Average token count per call looks like an efficiency metric but means little without segmentation. Your code assistant feature might average 8,000 tokens per call while your classification feature averages 200. An average of the two tells you nothing about either. Segment by feature and it becomes useful. Be careful comparing token counts across model generations, too: Anthropic says Claude 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text, with the exact increase depending on the content.

Model distribution is worth tracking if you route between models, but on its own it describes rather than prescribes. Knowing that 60% of your calls go to Claude Sonnet 5.5 and 40% to Claude Haiku 5.5 only matters once you attach cost and quality to each bucket. Sonnet 5.5 costs $2 per million input tokens and $10 per million output tokens, while Haiku 5.5 costs $0.10 and $0.50 for prompts up to 100,000 tokens, a twentieth of the price. Without cost and quality attached, the split is an inventory, not an optimization input.

How to Instrument This

Log one row per call with a consistent set of fields: a UTC timestamp, feature name, task ID, user ID, the model that answered, input tokens, cached tokens, cache write tokens, output tokens, reasoning tokens, latency in milliseconds, attempt number, whether the call succeeded, and a quality score where you have one. Keep a second small table with one row per task and whether it succeeded. This works in a time-series database, a data warehouse or a Postgres table with an index on the timestamp.

The token fields come straight from the usage object:

from datetime import datetime, timezone


def token_fields(provider: str, usage) -> dict:
    """Token counts for the metrics above, from the provider's usage object."""
    if provider == "openai":
        # Responses API: input_tokens already includes cached and cache-write tokens.
        d = usage.input_tokens_details
        cached = d.cached_tokens or 0
        written = getattr(d, "cache_write_tokens", 0) or 0
        total_input = usage.input_tokens
        reasoning = getattr(getattr(usage, "output_tokens_details", None), "reasoning_tokens", 0) or 0
    elif provider == "anthropic":
        # Messages API: input_tokens counts only tokens after the last cache breakpoint.
        cached = usage.cache_read_input_tokens or 0
        written = usage.cache_creation_input_tokens or 0
        total_input = usage.input_tokens + cached + written
        reasoning = getattr(getattr(usage, "output_tokens_details", None), "thinking_tokens", 0) or 0
    else:
        raise ValueError(f"no usage mapping for {provider}")
    return {
        "ts": datetime.now(timezone.utc).isoformat(),
        "input_tokens": total_input,
        "cached_tokens": cached,
        "cache_write_tokens": written,
        "output_tokens": usage.output_tokens,  # already includes reasoning tokens
        "reasoning_tokens": reasoning,
    }

For the cost of each call, use the cost function in LLM Cost Tracking: Per Request, Feature and User, which prices cache reads and writes per model, rather than keeping a second copy. With cost and these counts stored, three of the metrics above are one query:

SELECT c.feature,
       SUM(c.cached_tokens)::numeric    / NULLIF(SUM(c.input_tokens), 0)  AS cache_hit_rate,
       SUM(c.reasoning_tokens)::numeric / NULLIF(SUM(c.output_tokens), 0) AS reasoning_share,
       SUM(c.cost_usd) / NULLIF(COUNT(DISTINCT c.task_id) FILTER (WHERE t.succeeded), 0)
         AS cost_per_successful_task
FROM llm_calls c
JOIN tasks t ON t.task_id = c.task_id
WHERE c.ts >= now() - interval '7 days'
GROUP BY c.feature;

Build two views from this data. An operational view updates hourly and shows latency percentiles, error rate, retry rate and cache hit rate per feature, since one prompt change can drop the hit rate overnight. You look at it when something feels wrong. A financial view aggregates daily and shows cost per feature, cost per successful task, reasoning share, cost per output unit and the week-over-week trend. You look at it when planning optimization work.

Building all of this takes engineering time, which is why many teams put it off until costs already hurt. PromptUnit covers part of it. It checks each request and uses a cheaper model when one is just as good for that request; otherwise the request goes to the model you asked for. Setup is two settings: point your app's AI API base URL at PromptUnit and add your PromptUnit key, and your own provider key stays. The dashboard shows every request with the model asked for, the model that answered, the cost and the saving. It groups requests into the steps of your app or agent and shows what each step costs, and it shows cost per feature when you tag requests with the optional x-promptunit-feature header. Optional side-by-side checks, off by default and paid on your own provider keys within a daily budget you set, have cheaper models answer a sample of your real requests after the answer has gone back to your app, and the model you asked for judges whether each cheaper answer is as good. PromptUnit has no per-user view, so keep per-user numbers in your own logs.

Connecting Metrics to Decisions

The point of this stack is a clear path from a number to an action:

What you see	Where to look
High cost per call on one feature	Model choice or prompt design for that feature
Cost per successful task rising while cost per call falls	A cheaper model that fails, retries or takes more steps
Low token efficiency ratio	Prompt and context trimming
Low cache hit rate with many cache write tokens	Something changing near the start of the prompt
High reasoning share on a simple feature	The effort setting for that feature
High retry rate	Output validation or provider reliability
Low quality score on a cheap model	A stronger model for that task, or a better prompt

Each metric has a natural next step. Monthly spend does not.

Teams that build this instrumentation early find that the data reorders their engineering priorities. The feature you assumed was expensive turns out to be efficient. The feature you ignored turns out to cost three times what you thought. That reordering, based on numbers rather than assumptions, is what good LLM observability produces.

To see every request with the model you asked for, the model that answered, its cost and its saving, start free with PromptUnit. It is free until it has saved you money, then it takes 20% of what it saves.