Spend ten minutes watching a deep research agent work and you’ll see something strange. The model reads the question. It thinks. Then it pauses mid-thought, fires off a search query, scans what comes back, and keeps reasoning. Sometimes it searches again. And again. It looks less like software than like an analyst hunched over a terminal at 2 a.m.
That behavior is the most interesting thing happening in AI right now, at least for anyone who works on RAG pipeline design. OpenAI shipped Deep Research in February 2025 as an o3-based agent that browses for 5 to 30 minutes per question. Google built Deep Research into Gemini. Perplexity staked its product on it. Anthropic added a research mode to Claude. What started as a flagship feature is now table stakes.
Meanwhile, a quieter change happened in the research community. Teams stopped bolting retrieval onto models and started training it in. Search-o1, published by Li et al. in January 2025, lets a Qwen-based reasoning model call a search engine mid-chain through a literal <search> tag. Search-R1, from Jin et al., used reinforcement learning to teach Qwen models when to search, what to ask, and when to stop. R1-Searcher and DeepResearcher extended the approach.
The reported results point the same direction: models that decide when to search beat the same models running a fixed RAG prompt, often by double digits on open-domain QA benchmarks.
This breaks the core assumption behind most enterprise RAG pipeline design. We built pipelines around one shot. Query comes in, retrieve top-k chunks, stuff them in a prompt, generate an answer, done. Retrieval was a stage. Now it’s a decision the model makes, sometimes ten times in a single conversation.
That doesn’t mean you need to rip out your stack. Most of what changes is design, not plumbing. Five shifts in RAG pipeline design are worth making while the technology settles: treat retrieval as a tool, design for multi-hop by default, connect retrieval quality to inference cost, evaluate trajectories instead of only answers, and split your architecture into fast and deep lanes.
Work through all five and you should have a short list of changes you can make this quarter, plus the research to back them up when your team asks why.
Shift 1: Retrieval Becomes a Tool, Not a Stage
A classic RAG pipeline design has a fixed shape. Embed the query, pull the top 20 chunks, rerank, generate. Retrieval happens exactly once, in exactly one place, whether or not it helped.
Reasoning models break that shape. Search-o1 gave its reasoner a tool: the model emits a search request inside its chain of thought, gets results back, and continues reasoning over them. Search-R1 went further and trained the behavior with reinforcement learning, rewarding correct final answers and letting the retrieval policy emerge on its own. Nobody hand-wrote the rules for when to search. The model learned them.
What the change looks like in your architecture
Your retriever stops being a stage inside a pipeline and becomes a service the model calls on demand. A service needs things a pipeline stage never did:
- Sub-second latency, because an agent may call it ten times before answering.
- Metadata filtering, so the model can narrow scope itself: search only the 2024 vendor contracts, not the whole corpus.
- Versioned, stable ranking, because reproducible traces depend on consistent results.
- Citations returned with every result, since the model has to quote sources in its final answer.
Think of it like a database. Nobody calls Postgres “a stage in the pipeline.” It’s infrastructure that applications query when they need it. Retrieval is heading the same way, and that reframing is the heart of the new RAG pipeline design. Teams that make the switch early get a cleaner separation between their data layer and their model layer as a bonus.
Shift 2: Multi-Hop Is Now the RAG Pipeline Design Baseline
The questions deep research agents handle well are multi-hop by nature. “Did our biggest customer’s renewal include the new SLA, and how does that compare to what legal signed in 2023?” No single chunk answers that. It takes a chain.
Multi-hop has been measured for years. HotpotQA and 2WikiMultiHopQA test two-hop reasoning. MuSiQue piles on three and four hops with distractor documents mixed in. Bamboogle, a set of 125 questions introduced by Press et al. in 2022, was built specifically to catch systems that can’t chain facts. These benchmarks used to read like academic curiosities. They read differently now, because real users ask deep research agents these exact questions every day.
Why top-k fails on chains
For a boring reason. The question names the first hop, not the last one. Your embedding search returns chunks about the renewal and nothing about the 2023 legal terms. Traditional RAG pipeline design has nothing for that second hop. An agent can close the gap by searching again, but only if your index can answer the second query too.
Three fixes that carry the weight
- Hybrid retrieval. Dense vectors plus BM25 keyword search catches exact terms like contract IDs that embeddings blur. Anthropic’s contextual retrieval work is the best public data point here. Prepending short context descriptions to each chunk cut the top-20 retrieval failure rate by 35%. Adding contextual BM25 brought the reduction to 49%. Reranking took it to 67%.
- GraphRAG for entity-heavy corpora. Microsoft’s method links entities across documents, so a query about a customer can walk to their contracts, renewals, and signers without guessing the right phrasing for each hop.
- Query chaining in the agent itself, which works only if shifts 1 and 3 are already in place.
One more finding worth knowing. Liu et al.’s “Lost in the Middle” study, published in TACL in 2024, showed that models answer worse when relevant information sits in the middle of a long context, sometimes by more than 20 percentage points. In a one-shot pipeline you control chunk order with a reranker. When an agent stacks results from four searches into its own context window, you don’t. Keep result sets small and well ranked. The model will stack several of them.
Shift 3: Retrieval Quality Now Shows Up on Your Inference Bill
Test-time compute changed the economics of a wrong retrieval. In old-school RAG pipeline design, bad retrieval means a bad answer. Annoying, but bounded. In an agentic loop, a bad first retrieval means the model searches again, reads more documents, re-reasons over all of it, and burns tokens the entire way. You get billed for weak retrieval at thinking-token rates.
The new cost math
Token prices keep falling. The Stanford AI Index 2025 report found that inference costs for GPT-3.5-level performance dropped more than 280-fold between November 2022 and October 2024. DeepSeek’s R1 paper reported training costs around $5.6 million, which challenged the assumption that frontier reasoning needs frontier money. Reasoning is getting cheaper every quarter.
Cheap is not free. A deep research run thinks for 5 to 30 minutes. That’s easily 50 times the tokens of a single-shot RAG answer. A model that costs 10 times less per token but spends 100 times more tokens per answer still costs 10 times more overall. The efficiency gains only materialize when retrieval keeps the loop short. Retrieval quality now controls answer quality and inference cost at the same time.
Three levers that pay for themselves
- Reranking. If a reranker saves one search round on 20 percent of deep queries, it cuts the thinking tokens on those queries by a large fraction. Anthropic’s 67% failure reduction with reranking is the number to remember.
- Caching. Deep research agents ask overlapping questions. Embedding caches, result caches, and reuse across turns all reduce spend, and none of them change an answer.
- Loop caps. Set a maximum number of search rounds per query and a token budget per run. Search-R1’s models learned when to stop because the reward came from correct answers, not exhaustive ones. Your production system needs the same guardrail in code.
Shift 4: Evals Move From Answers to Trajectories
If your eval suite only answers “was the final answer good,” it will miss most of what changed. The interesting failures happen mid-run now. The model searched for the wrong thing. It searched three times when once would do. It found the right document and ignored it.
Search-R1 trained with outcome rewards: correct answer, positive reward, and the retrieval policy emerged as a side effect. That works for training. It doesn’t work for debugging production, because a correct answer reached through a wasteful path is still a problem you’re paying for.
Two layers of measurement
Keep the answer layer. Faithfulness, answer relevance, and context precision still catch hallucinations and poor grounding, and tools like RAGAS and DeepEval cover them well.
Add a path layer. The newest job in RAG pipeline design is measuring how the model got there. For each run, log:
- Search calls per query, and whether the average is trending up.
- Precision at each hop: of the documents returned in round two, how many got cited?
- Tokens spent per hop and total thinking tokens per resolved query.
- Stop behavior: did the model answer after the first good result, or keep going?
Replayability is the real prize
A trajectory suite is a set of recorded runs you re-execute when the model, prompt, or index changes. Without replays, every upgrade is a guess. With them, you can diff behavior: same question, new model, three fewer searches, same answer. That comparison is what makes upgrades easy to approve.
Anchor the suite in multi-hop sets. Bamboogle’s 125 questions and MuSiQue’s composition-heavy sets are small enough to run on every change. A pipeline that scores well on single-hop QA and falls apart on two-hop questions is exactly the regression you want to catch before users do.
Shift 5: Split the Stack Into Fast and Deep Lanes
Not every query deserves an agent. “What’s our refund policy for EU customers?” is a lookup. Sending it through a five-minute reasoning loop wastes money and tests the user’s patience.
Route instead. Most production traffic, often 70 to 90 percent, is single-hop: factual lookups, summaries, document QA. A classic RAG path answers those in under a second at a fraction of the cost. The deep lane, an agentic search loop, handles the rest: comparisons, multi-source synthesis, anything with “and how does that compare” in it. This last shift in RAG pipeline design is mostly about restraint.
Routing without overthinking it
The router itself is small. A classifier prompt on a cheap model, a few rules about question structure, even heuristics like query length and connective words can route most traffic correctly. Start with simple rules, measure how often they misroute, then improve.
Two details that make the split work
First, different service levels. The fast lane needs p95 latency under a second. The deep lane needs progress indicators and a background job model, because users tolerate minutes of research only when they can see something happening. Second, a shared index. Both lanes should hit the same retrieval service with the same ranking logic, so improvements help everywhere and traces stay comparable.
This is also why the “RAG is dead” takes keep aging badly. The fast lane is RAG, unchanged. The deep lane is RAG with a loop wrapped around it. What died was the assumption that one architecture serves every question.
What to Do This Quarter
The interesting story in AI right now isn’t the death of retrieval. The model walked into the middle of the pipeline and started making decisions. Deep research agents pause, search, read, and search again because training methods like Search-R1’s turned retrieval into a learned behavior instead of a hardcoded step.
Five RAG pipeline design shifts follow from that. Retrieval becomes a callable service. Multi-hop becomes the design baseline. Retrieval quality becomes a cost lever. Evals grow a second layer that scores paths, not just answers. The stack splits into a fast lane for lookups and a deep lane for research.
You don’t need all five this month. Start with the first one, because everything else in RAG pipeline design builds on it: wrap your retriever in a clean API with metadata filters, citations, and sub-second latency. Then instrument search calls and tokens per query so you can see what your traffic looks like. The data will tell you which of the other four to tackle next.
We track these shifts as they land, with the benchmarks and cost math attached. Subscribe to the Rag About It newsletter for weekly breakdowns, or send us your RAG pipeline design and we’ll point out where the reasoning model wave will hit it first.
