“Can we just paste the whole knowledge base into the prompt?”
If you build enterprise RAG systems, you’ve heard some version of that question in a planning meeting. It used to be easy to answer. Context windows topped out around 128K to 200K tokens, so anything bigger than a small codebase forced you into retrieval whether you liked it or not.
Now the RAG vs long context question deserves real math. The “RAG is dead” debate flared up again this week across LinkedIn and X, the way it does every time a frontier lab ships a bigger window. And the windows are genuinely big now. OpenAI’s GPT-4.1 lists a 1 million token window at $2 per million input tokens. Anthropic opened a 1 million token beta for Claude Sonnet and charges a premium above 200K. Google has shipped 2 million token windows since Gemini 1.5 Pro. Meta’s Llama 4 Scout claims 10 million. Add prompt caching, which cuts the cost of repeated context by 75 to 90 percent, and you can see why executives keep asking whether retrieval is still worth building.
The honest answer is that both camps argue about the wrong number. Capability stopped being the question around the time windows crossed a million tokens. Cost per query, latency, cache behavior, and auditability are the questions now, and those have measurable answers. RAG survived its first decade for a reason the original paper from Meta AI (Lewis et al., 2020) named plainly: parametric knowledge goes stale and can’t cite its sources.
So this post runs the arithmetic. We’ll use public rate cards, name the failure modes the hot takes skip, and end with five numbers you can carry into your next budget review. One of them is a per-query cost gap of about 80x. Another is a compliance deadline that already passed in August. No vibes, just invoices.
What Actually Changed
Two shifts, not one. Windows got huge, then caching made huge look affordable. Only the first shift is what it seems.
Million-token windows went mainstream
The numbers come straight from the providers’ rate cards:
- GPT-4.1 (April 2025): 1 million token context, $2 per million input tokens.
- Claude Sonnet: 1 million token context in beta, with surcharged pricing above 200K tokens.
- Gemini: 2 million token windows since the 1.5 Pro release.
- Llama 4 Scout: a listed 10 million token window, which deserves an asterisk. Independent tests rarely find full quality across a model’s advertised span, and we’ll get to the benchmark for that.
A million tokens is roughly 750,000 words. Call it 15 novels. Entire mid-size knowledge bases, policy libraries, and contract repositories now fit in a single prompt. Technically, “just paste it” works.
Prompt caching changed the bill, partly
The second shift matters just as much. OpenAI prices cached input at about a quarter of the standard rate. Anthropic charges roughly a tenth of the base input price for cache reads, plus a 25 percent premium on writes. Google sells explicit context caches and bills storage by the token-hour.
Caching is a real discount for static content. The catch, and it’s a big one, is that caching rewards prefixes that never change. Your knowledge base is not that. Hold that thought until number three below.
5 Numbers That Settle RAG vs Long Context
1. $1.60 versus two cents, per query
Say your knowledge base is 800,000 tokens. At GPT-4.1’s list price of $2 per million input tokens, stuffing it into every prompt costs $1.60 per query, before a single output token.
The RAG version: embed the question, retrieve the 10 best chunks, and pass those along. That’s roughly 8,000 retrieved tokens plus a 1,000 token query and system prompt, so 9,000 tokens at the same rate comes to $0.018. Call it two cents.
The gap is about 80x per query. Scale it to 5,000 queries a day across 250 working days and long context costs about $2 million a year. Retrieval costs about $25,000. Model prices will keep falling, but a price cut applies to both sides, so the ratio survives every rate card update.
On Claude Sonnet at $3 per million input tokens, the same math gives $2.40 versus three cents. Different invoice, same 80x. Every serious RAG vs long context comparison starts from that ratio.
“Caching fixes this,” the optimists say. A warm read drops the 800K prompt to roughly $0.16 on Anthropic’s cache rate, $0.40 on OpenAI’s. But that assumes the cache is warm, the corpus hasn’t changed, and the TTL hasn’t expired. That’s number three.
2. Seconds versus milliseconds
The second RAG vs long context number is time. Before a model writes anything, it has to read everything you sent. Prefill compute scales with input length, so an 800,000 token prompt adds seconds before the first word appears. Anthropic charges extra above 200K tokens for exactly this reason: long prompts cost more to serve, and the rate card says so.
Support and product bots usually run on a three-second response SLO. Retrieval’s overhead is a vector lookup, typically 10 to 50 milliseconds at enterprise index sizes, plus prefill over 9,000 tokens instead of 800,000. The user gets an answer before the long-context setup finishes loading its prompt.
3. Cache hit rates collapse on live corpora
Cache behavior is where RAG vs long context comparisons get slippery, because caching needs an identical prefix. Live knowledge bases break that assumption from several directions at once:
- One price sheet updates Monday morning and the cached prefix goes stale. Refresh it and you pay the write premium again.
- Anthropic’s default cache lifetime is five minutes. Bursty traffic hits cold caches anyway.
- Google bills cached tokens by the hour whether queries arrive or not. Parking a 2 million token corpus in cache all month is a real line item.
- Document-level updates mean the shared prefix your whole caching plan depends on changes daily.
Static content genuinely benefits: model cards, system prompts, regulation texts. A nightly-updated enterprise corpus is the opposite case, and it’s the case most enterprises are actually in.
4. Claimed context is not effective context
Effective context is the RAG vs long context number vendors don’t print. NVIDIA’s RULER benchmark (Hsieh et al., 2024) tested long-context models on retrieval and reasoning tasks across their advertised spans and found effective context well below the claimed figures, sometimes by a wide margin. Advertised windows tend to come from easier needle-in-a-haystack tests where a single fact hides in filler.
Placement matters too. “Lost in the Middle” (Liu et al., 2023, published in TACL) showed accuracy follows a U-shaped curve: models use the start and end of a prompt best and lose material in the middle. An 800K token dump guarantees your key facts spend time exactly where the model reads worst.
Retrieval is a placement strategy. It puts the ten most relevant chunks at the front, in full, where attention is strongest. For multi-hop questions, ten chunks read carefully beat a million tokens skimmed.
5. Freshness, and the audit trail nobody prices in
The last two RAG vs long context costs never appear in a demo.
The first is freshness. Enterprise corpora change hourly: tickets, policies, prices, product docs. RAG pays a small re-embedding fee for changed chunks. Long context pays full input price again, every query, forever, because the only way to update the prompt is to resend it.
The second is provenance. Retrieval logs record which chunk, which score, and which source document produced each answer. That log matters now in a way it didn’t two years ago. The EU AI Act’s general-purpose AI obligations took effect in August 2025, and its high-risk regime began applying in August 2026. When a compliance team asks “which source produced this sentence?”, a retrieval log answers in one query. A million-token blob has no clean answer at all.
Where Long Context Genuinely Wins
None of this says big windows are a gimmick. They’re a real upgrade for the right jobs, and the RAG vs long context call isn’t one-sided.
Three cases to skip retrieval
- Small corpora. Under roughly 50K tokens, pasting beats building. One product spec, one contract set, one small codebase: prompt it and move on.
- One-off, high-value tasks. A model reading a full 300-page filing once, even at $2 to $3, costs less than an analyst hour and finishes faster.
- Whole-document reasoning. Some tasks are about the document as a whole: structure, cross-references, global themes. Retrieval shreds exactly what those tasks need kept intact.
If your workload looks like any of these, the “just paste it” crowd is right. Take the win.
The hybrid most teams land on
In production, the RAG vs long context decision usually lands on a hybrid. Retrieve to select, then read in full: pull 50 candidates, pass the best 10 chunks into a long-context model so it sees complete text instead of truncated fragments. The big window improves retrieval quality, since you can afford longer chunks and less aggressive compression.
Agentic setups push further. The model decides when it needs more context, pulls it on demand, and reads it whole. We covered those agentic search patterns in an earlier post; the short version is that long context made the retrieval step more useful, not less used.
Long context didn’t kill RAG. It upgraded RAG’s reading step.
Run the Math on Your Own Stack
Twenty minutes and five numbers. Here’s how to run the RAG vs long context numbers where you work:
- Queries per day, from your product or support analytics.
- Corpus size in tokens, from your ingestion pipeline logs.
- Per-query context cost: corpus tokens times your model’s input price.
- Per-query RAG cost: retrieved tokens times input price, plus embeddings. At OpenAI’s text-embedding-3-small rate of $0.02 per million tokens, embedding costs round to zero.
- A stress-tested cache assumption: what fraction of queries hit a warm cache if the corpus changes daily? If your honest answer is “half,” halve the long-context bill and compare again.
A worked example. Two-million-token knowledge base, 50,000 queries a month, $2 per million input tokens. Long context: $4 per query, or $200,000 a month. RAG: about $0.018 per query, or $900, plus a few hundred dollars of embedding and vector search hosting. At that volume the long-context line item reaches the board, and retrieval looks like a rounding error.
Once it’s live, track four metrics: cost per resolved query, cache hit rate, precision@10, and your own needle tests for effective context. Rising precision and falling cost per resolved query mean the architecture is working, whatever the hot takes say.
Across all five numbers, the RAG vs long context pattern is the same. Long context is real, getting cheaper, and worth using. It loses on the per-query invoice, the latency budget, cache fragility, the gap between claimed and effective context, and provenance under audit. So the next time someone asks whether you can just paste the whole knowledge base into the prompt, say yes, and hand them the invoice for their query volume.
We built Rag About It for exactly these conversations. Subscribe to the newsletter at ragaboutit.com for weekly breakdowns like this one, and if you’re weighing the retrieval side of the decision, our guides on RAG data freshness and access control walk through the build. Bring your numbers. The debate flares up every time a lab ships a bigger window. Your invoice won’t care.
