The Hidden Costs in Your RAG Pipeline (And How to Cut Them)
Retrieved context alone can cost $12,000 a month at 100,000 RAG calls a day. The four real cost centers, how to filter passages, and what each step costs.
At 100,000 LLM calls per day with Claude Sonnet 5.5, a naive RAG implementation that retrieves 10 chunks of 200 tokens each adds 200 million input tokens to your daily spend. At $2.00 per million input tokens (Anthropic's price in October 2026), that is $400 per day in retrieved context alone, or roughly $12,000 per month. That cost has nothing to do with the quality of your LLM responses. It is pure overhead from a retrieval strategy that pulls too much context and does not compress it before sending. Most teams building RAG systems do not see this because they account for LLM costs as a single line item, not broken down by where the tokens are coming from.
How do I filter RAG passages to reduce cost? Retrieve a wider candidate set with vector search, rerank it, keep the top 3 to 5 passages above a relevance score cutoff, drop near-duplicate chunks from overlapping windows, and stop adding passages once a fixed token budget is full. Going from 10 chunks of 200 tokens to 4 cuts the retrieved context by 60%: in the example above, from about $12,000 to about $4,800 a month. The section on filtering below walks through each step, and the rest of this post covers the other three places a RAG pipeline spends money.
RAG pipelines have four distinct cost centers. Treating them as one makes it nearly impossible to optimize any of them.
Cost Center 1: Embedding Calls
Embedding costs are often dismissed as negligible, and at small scale they are. At text-embedding-3-small pricing ($0.02 per million tokens), a one-time indexing job on 1 million chunks of 200 tokens each costs $4.00. That is cheap. The ongoing query cost at 10,000 users per day, each sending a 50-token query for embedding, is 500,000 tokens per day, or about $0.01 per day. Not worth optimizing.
The embedding cost problem surfaces in two scenarios. First, if you re-embed your document corpus frequently, because documents are updated often or because you are experimenting with different chunking strategies, the indexing cost can grow significantly. If you are re-embedding a 50 million token corpus weekly, that is $1.00 per run with text-embedding-3-small and $6.50 per run with text-embedding-3-large ($0.13 per million tokens), and the bill grows with every chunking experiment you re-run. Second, if you are using embeddings as part of a multi-stage retrieval pipeline that embeds intermediate results or reformulated queries, the per-query embedding cost multiplies by the number of stages.
The fix is simple: cache embeddings. Document chunk embeddings are deterministic. If the chunk content has not changed, the embedding has not changed. Storing embeddings in a vector database with a content hash as the key means you only compute new embeddings when documents actually change. This is so obvious it feels redundant to say, but a surprising number of pipelines re-embed on every ingest without checking for existing embeddings.
Cost Center 2: Context Window Bloat
This is the dominant cost driver in most production RAG systems, and it is the one most directly under your control.
The standard retrieval pattern fetches the top-k most semantically similar chunks to the user query. A common default is k=10, with chunks of 200-300 tokens each. That adds 2,000 to 3,000 tokens of retrieved content to every LLM call, on top of your system prompt, the user query, and any conversation history. At high call volume, this compounds fast.
Consider the math more precisely. If your system prompt is 500 tokens, your user query averages 100 tokens, and you retrieve 10 chunks at 250 tokens each (2,500 tokens), your total input per call is 3,100 tokens. If you cut retrieval from 10 chunks to 3 chunks (750 tokens), your total input drops to 1,350 tokens. That is a 56% reduction in input tokens per call, which directly translates to a 56% reduction in input token cost. On a 100,000 call per day workload with Claude Sonnet 5.5, the 1,750 tokens saved per call come to 175 million input tokens a day: $350 per day, or about $10,500 per month, from changing a single integer in your retrieval configuration.
The risk of reducing k is that you retrieve fewer relevant chunks and degrade answer quality. This is where reranking becomes relevant.
Cutting k is not the only lever, though. For some workloads the better fix is skipping retrieval altogether in favor of a long-context model with prompt caching. A 1M-token context window at a low price, such as Gemini 3.5 Flash-Lite's at $0.30 per million input tokens, changes the RAG-versus-long-context tradeoff for document-scoped chat and corpus-wide synthesis queries, though it is not a universal replacement for retrieval on high-volume, many-query workloads.
Paying full price on every request?
PromptUnit checks each request your app sends and moves it to a cheaper model when one is just as good for that request. You change two settings, and your own provider key stays. Free until it has saved you money, then 20% of what it saves.
Try it free →Cost Center 3: Reranking
A reranker is a cross-encoder model that takes your retrieved candidates and scores them for relevance to the specific query, more accurately than embedding similarity alone can. Services like Cohere Rerank are purpose-built for this. Reranking 10 candidates to identify the top 3 costs a small amount per query, but it allows you to retrieve a larger initial candidate set with cheap vector similarity search (which is nearly free) and then aggressively filter to a small, high-relevance context window.
The economics: reranking costs money, but sending 3 high-quality chunks instead of 10 medium-quality chunks to your LLM saves substantially more in input token costs. At scale with expensive models, the reranking cost is typically 10-20% of the token savings it enables.
There is a secondary quality benefit: smaller, more relevant context windows often produce better outputs. Models can focus on fewer, more relevant passages rather than synthesizing across 10 chunks of varying relevance. This means lower hallucination rates and better grounding, which reduces downstream costs from bad outputs requiring regeneration.
How to Filter RAG Passages Before They Reach the Model
Filtering is where most of the context cost comes back. Five steps, in the order they usually pay off:
- Treat k as a maximum, not a target. Keep a chunk only if its similarity score clears a floor you set from your own data. A narrow question often has two good passages, not ten.
- Rerank a wider set, keep a narrow one. Pull 20 to 30 candidates with vector search, rerank them, and keep the top 3 to 5 above a reranker score cutoff.
- Drop near-duplicates. Overlapping chunk windows from the same document often carry the same sentences twice. Keep the higher-scoring copy.
- Fill a token budget in score order. Set a context budget, for example 1,500 tokens, add passages from the best down, and stop when it is full. Cutting an average of 2,500 retrieved tokens to 1,500 saves 100 million input tokens a day at 100,000 calls, or $200 a day at Claude Sonnet 5.5's input price.
- Trim long passages to the sentences that match the question. When chunks are long, an extractive step that keeps only the matching sentences cuts tokens without paraphrasing anything away.
Measure before and after on a held-out set of real questions: the retrieved tokens per call and the answer quality. If quality drops, raise the budget or the number of passages kept before touching anything else.
Cost Center 4: Multi-Round Retrieval
Some RAG architectures perform multiple retrieval and generation cycles per user query. A query decomposer breaks a complex question into sub-questions, each sub-question triggers a retrieval round, intermediate answers are synthesized, and a final generation pass produces the response. Each round is a full LLM call with a full context window of retrieved content. A 3-round pipeline costs roughly 3x what a 1-round pipeline costs at the same retrieval settings.
Multi-round RAG can significantly improve answer quality on complex, multi-faceted questions. The question is whether the quality improvement justifies the cost multiplier for your specific use case. A good benchmark is to run both pipeline configurations on a representative sample of real queries from your application, score the outputs on your quality criteria, and then calculate the cost-per-quality-point tradeoff. For many SaaS applications, the majority of queries are simple enough that single-round retrieval with a good reranker matches or exceeds multi-round quality at a fraction of the cost.
If you need multi-round retrieval for complex queries, route only those queries to the multi-round pipeline. Use a cheap classifier, such as GPT-6 Luna or Claude Haiku 5.5 (both $0.10 per million input tokens in October 2026), to score query complexity and decide which pipeline to invoke. Simple queries go to the single-round path, complex queries go to multi-round. This is the same model routing logic described in the LLM model routing guide, applied to pipeline architecture rather than just model selection.
Cutting All Four Simultaneously
The compounding opportunity is real. A team that implements all four optimizations: embedding caching, reduced k with reranking, and single-round routing for simple queries, can cut RAG infrastructure costs by an estimated 60 to 80% with little loss in quality. That is our estimate, not a measurement. The exact savings depend on your current configuration, model choices, and query distribution.
Context compression is another lever worth implementing once you have the other optimizations in place. Before inserting retrieved chunks into the LLM context, run them through a compression step that removes redundant sentences, extracts only the most query-relevant spans, and truncates to a maximum token budget. This is not summarization, which would lose information, but structural compression that removes tokens that are not load-bearing for the query at hand. This step belongs in your own pipeline: PromptUnit sends the context you retrieve as written, and works on the model side instead, using a cheaper model when one is just as good for a request.
The interaction between RAG cost optimization and model choice is also worth considering carefully. If you are running RAG on Claude Sonnet 5.5 ($2/$10 per MTok) or a larger model for tasks that do not require its full capability, routing to a cheaper model for simple retrievals can compound the savings from context compression. For a complete view of model selection tradeoffs, see the Anthropic Claude API pricing guide alongside the general LLM model routing guide.
Start with cost center 2. Measure your average retrieved token count per call today. Cut k from whatever it is to 3-5, add a reranker if you do not have one, and measure quality on a held-out test set. The context window reduction is almost certainly your fastest path to real cost savings.
To see what each step of your pipeline costs in production (query rewriting, reranking with an LLM, the final answer), PromptUnit shows every request with the model asked for, the model that answered, what it cost and what it saved, and groups requests into the steps of your app. It checks each request and uses a cheaper model when one is just as good for it. Setup is two settings: the API address and your PromptUnit key.