The most interesting story in AI this week isn’t a new benchmark score. It’s the spread of deep research agents into products your users already open every day.
Open ChatGPT’s deep research mode and ask it something hard. Something like “which vector database best handles filtered hybrid search at 10 million documents?” Then step away for a while. Twenty minutes later you get a multi-page report with comparison tables and citations you can click and check.
That wait is the product working. OpenAI says deep research spends five to thirty minutes per question, running searches, reading what comes back, and re-searching when the first pass looks thin. Google’s Gemini Deep Research works the same way. So does Perplexity, and Claude’s web search on a smaller scale.
Now ask a typical enterprise RAG (retrieval augmented generation) stack the same question. You get an answer in under three seconds. One vector search, maybe a rerank, five chunks into the context window, then generate.
The speed is real. The accuracy often isn’t.
This gap is worth studying because the consumer labs already ran the experiment for you. Deep research agents treat retrieval as a process: plan, search, read, verify, repeat. Most enterprise systems still run the 2022 blueprint: embed, store, retrieve top-k, generate. Then the team wonders why the system fumbles multi-hop questions, invents policy details, and cites last year’s document.
The shift also raises the bar for internal tools. Your users have seen citations attached to every sentence. They’ve watched an agent admit a source was thin and search again. Somewhere around the third hallucinated footnote, “the model said so” stopped being an acceptable answer at work.
You can’t copy the twenty-minute latency. Nobody wants a benefits chatbot that takes a coffee break. But the retrieval logic behind these deep research agents is public, in papers and product writeups from OpenAI, Google DeepMind, Meta, and Anthropic, and it translates cleanly to enterprise RAG. Five lessons stand out: plan before you search, retrieve in loops, verify before you answer, cite at the chunk level, and score your sources. Here’s each one, with benchmarks and tools you can apply this week.
Lesson 1: Plan the Query Before You Search
What deep research agents do first
They plan. OpenAI’s deep research breaks a question into sub-questions and decides what evidence each one needs before it runs a single search. Perplexity rewrites your input into a cleaner, more searchable query before retrieval starts. The search itself is almost the afterthought.
Your users do the same thing to your RAG system, minus the planning. They type compound questions. “What was our Q3 churn in EMEA and how did it move against Q2?” is two retrievals wearing one sentence. A single vector query on that text pulls half of one answer and half of another, ranked by surface similarity.
The MultiHop-RAG benchmark, released in 2024, measures exactly this failure. On questions that need evidence from multiple documents, the strongest setup scored around 40 F1, a steep drop from its single-hop numbers. Retrieval wasn’t the bottleneck. The single-pass design was.
How to add the planner step
Run one cheap LLM call before retrieval. It splits the question into sub-queries, attaches metadata filters to each, and sends them as separate searches. For the churn question, the planner should emit two clean lookups: “EMEA churn, Q3, revenue reports” and “EMEA churn, Q2, revenue reports.” Compare the two in generation, not in the vector store.
Small models handle this fine. Gemini Flash or Claude Haiku will split a query for a fraction of a cent. LlamaIndex ships the pattern as its SubQuestionQueryEngine, and LangGraph gives you a planner node where you control the logic yourself. It’s an afternoon of work, and it fixes the questions that currently get escalated to your Slack channel.
Lesson 2: Deep Research Agents Retrieve in Loops, Not Lookups
Why one shot fails
Single-shot retrieval assumes the answer lives in the first pass. Deep research agents assume the opposite. They search, read, notice what’s missing, and search again. OpenAI’s deep research runs multiple searches per report by design, and Claude’s web search follows up on pages as it reads them.
Why not just retrieve more chunks in one shot? Because context volume isn’t context quality. The Lost in the Middle study (Liu et al., 2023) found that models use the beginning and end of the context window far better than the middle. Feed twenty retrieved chunks and the one that matters can sit at position eleven, ignored. Refining relevance beats adding volume.
Two patterns to copy
Corrective RAG (Yan et al., 2024) inserts an evaluator that grades retrieved documents before generation. Low grades trigger a fallback, like a web search or a second index. Self-RAG (Asai et al., 2023) goes further and trains the model to critique its own retrieval decisions, and it outperformed ChatGPT on several open-domain QA benchmarks while doing it.
Your enterprise version doesn’t need to be open-ended. Cap it at two or three rounds. Retrieve, ask the model “does this evidence answer the question?”, and run a refined query if the answer is no. Most questions close in one round. The hard ones, the ones that get escalated anyway, get the extra pass and two extra seconds of latency.
LangGraph state machines handle the loop naturally, and LlamaIndex Workflows does the same with an event-driven design. Start with a sufficiency prompt and a hard cap before you build anything fancier.
Lesson 3: Verify Before You Answer
The verification pass
Deep research agents spend a big chunk of those twenty minutes checking. They draft, test the draft against sources, and rewrite whatever doesn’t hold up. Meta’s chain-of-verification technique (Dhuliawala et al., 2023) showed the pattern works with any decent model: draft an answer, generate verification questions from your own claims, answer them independently, then revise. Factuality improved across the tasks the authors tested.
Your stack can run the same pass at small scale. After the generator drafts an answer, a second LLM call checks each claim against the retrieved chunks. Unsupported claims get flagged or dropped. Supported ones get their citations attached. That’s the whole technique.
Tools and cost
The tooling here is mature. RAGAS measures faithfulness, the share of claims grounded in your retrieved context. TruLens scores groundedness the same way. Arize Phoenix traces each pipeline step, so when an answer goes wrong you can see exactly where. Pick one and wire it into CI, so every prompt or index change gets scored before it ships.
Cost is the obvious objection, and it’s manageable. Run verification on the answers that carry risk: compliance lookups, policy questions, anything with legal exposure. Skip it for casual chat. A small model does the checking for fractions of a cent, which is a lot cheaper than a confidently wrong answer about PTO policy reaching a new hire.
Lesson 4: Cite at the Chunk Level
Why citations are the product
Every consumer research agent cites inline. Perplexity attaches sources to sentences. Claude’s web search and Gemini’s Deep Research do the same. That isn’t decoration. Citations are how deep research agents earn trust, and how users catch errors before those errors spread into a doc, a ticket, or a reply to a customer.
Enterprise RAG needs the same discipline, and Anthropic’s contextual retrieval technique is the reference implementation. Before embedding, prepend each chunk with document context: the title, a one-line summary, and what the chunk specifically covers. Chunks stop being orphaned text fragments and start carrying their provenance into the vector store.
The numbers are worth reading twice. Anthropic reported that contextual embeddings cut top-20 retrieval failures by 35%. Contextual BM25 cut them by 49%. Combined, 67%. All from a preprocessing step, with no new model involved.
Wire it through the interface
Then close the loop in the UI. Store chunk IDs through the whole pipeline, prompt the generator to reference them inline, and render them as clickable footnotes. When a user clicks a citation, they should land on the exact paragraph, not on page 40 of a PDF.
Chunk-level citations change debugging too. When someone reports a wrong answer, you can trace which chunk backed which claim and fix the retrieval problem at its source. Aggregate citations hide the failure. Sentence-level citations expose it.
Lesson 5: Score the Sources, Not Just the Chunks
Deep research agents don’t rank results on embedding similarity alone. They weigh recency, corroboration, and whether a source is primary. A 2019 blog post and a 2026 official spec both “match” a query about the same feature. Only one should win.
Your index has the same problem, usually worse. Outdated policies, duplicate versions, abandoned drafts. Vector similarity scores all of them identically, so your RAG system will cite a policy that was replaced last quarter, confidently, because nothing told it the newer one exists. We covered the slow-burn version of this problem in our post on index rot. Source decay is the same disease with different symptoms.
Fixes that fit in a sprint
- Stamp every document with ingestion date, effective date, and status (draft, current, superseded) as metadata, not as body text
- Filter on recency by default, and make older documents an explicit opt-in
- Boost authoritative sources in ranking: official docs over wikis, signed policies over meeting notes
- Run a quarterly audit that flags anything untouched for over a year
Google DeepMind’s FRAMES benchmark shows why the effort pays. Gemini 1.5 Flash scored 57.6% on factuality questions without retrieval and 66.6% with web search attached. Retrieval helps, but the ceiling is set by what you feed it. A stale index makes every downstream step worse, and no reranker fixes that.
Where to Start
The twenty-minute report and the three-second answer aren’t different products. They’re different bets about where to spend effort. Deep research agents put their budget into planning, looping, verifying, citing, and filtering before generation. Classic RAG puts it all into generation speed.
Five lessons, one theme: retrieval is a process, not a lookup.
You don’t need to ship your own deep research agents next sprint. Add a decomposition step first. Then a sufficiency check in your retrieval loop, then faithfulness scoring in CI. Those three changes close most of the gap between your stack and the consumer products your users quietly compare it against. Chunk-level citations and source metadata can follow the sprint after.
If you want the architecture deep dive, read our guide on the RAG failure modes agentic designs fix. And if breakdowns like this one are useful, subscribe to the newsletter below. One RAG engineering topic per issue: benchmarks, tools, patterns, and the research you’d otherwise spend a Friday afternoon digging up yourself.
