Somewhere in a Slack channel this week, a product manager asked the question every retrieval team is starting to field: “Why do we still run our own vector database?”
The team had spent six months building a RAG stack. Chunking rules, embedding model swaps, a reranker, eval scripts, dashboards. Then their model provider shipped document search directly in the API, with citations, priced per query. The PM wasn’t trying to be difficult. The feature genuinely covers most of what that stack does, with no infrastructure to patch on weekends.
That question is the most interesting story in AI right now, at least for anyone who builds retrieval for a living. The R in RAG is moving into the model layer. OpenAI serves web search and file search through its Responses API. Anthropic gave Claude web search, a Files API, and connectors to external systems. Google grounds Gemini responses in Search. Cohere designed its Command models around retrieval with inline citations from the start. Capabilities that used to be a quarter-long build now appear as a line item on someone else’s pricing page.
The shift is fast and it’s awkward. Fast, because providers are competing on retrieval as a selling point, and every release pressures the rest. Awkward, because it invalidates planning assumptions. Budgets approved in January assumed nobody else would do this work. Plenty of those assumptions didn’t survive the year.
Most coverage misses the real decision, though. Native retrieval is not a verdict on your architecture. It’s a boundary that needs drawing, deliberately, between what the provider does well and what you must keep. Draw it by accident and you’ll either maintain a pipeline nobody uses or rip out plumbing that quietly enforces your security model.
So let’s get concrete about what model-native retrieval covers today, where it wins on speed and cost, where it breaks at enterprise scale, and the five decisions worth making before anyone touches your RAG stack.
What Model-Native Retrieval Actually Covers Now
“Built-in retrieval” gets used for three different things, and confusing them causes bad decisions. Separate them first.
Web grounding
The model queries a live search index and cites what it finds. OpenAI launched ChatGPT search in October 2024, then shipped web search into the Responses API in 2025. Anthropic added web search for Claude in February 2025. Google offers Gemini grounding with Search, which attaches sources from Google’s index to model responses. It handles factual, current, public questions well. It does nothing for anything sitting in your private corpus.
File search
You upload documents to provider-managed storage, and the API handles chunking, embeddings, retrieval, and citations. OpenAI’s file_search tool and Anthropic’s Files API both work this way. The provider stores your files, builds the index, and returns grounded answers with source snippets. This is the capability that made our hypothetical PM ask hard questions. For small corpora, it works, and it works this afternoon.
Connectors
The provider reaches into your systems through standardized integrations. This is where the Model Context Protocol matters. MCP started as Anthropic’s open spec in late 2024, and within about a year OpenAI, Google DeepMind, Microsoft, and AWS had all adopted it. Connectors stopped being bespoke integrations and became a shared surface. When a model can pull from your Drive, your CRM, or your wiki through a standard protocol, the retrieval layer is effectively the provider’s.
Why it landed in a rush
Two forces compressed the timeline. First, consumer pressure: Perplexity made cited answers a buying criterion, and every assistant vendor followed within months. Second, enterprise pull: buyers wanted grounded answers, not confident fiction. Menlo Ventures’ enterprise AI research tracked RAG adoption climbing from 31% of enterprise LLM use cases in early 2024 to 51% by the end of that year, and flagged hallucinations as a top concern in the same body of work. Providers read that demand and responded by absorbing retrieval into the product.
Where Native Retrieval Wins
The honest case for native retrieval is strong, and pretending otherwise doesn’t help anyone.
Time to first answer is the big one. A cited Q&A system over your PDFs, working in an afternoon. No chunking debates, no embedding model selection, no vector database to patch on Sundays. For a ten-person company, that difference is the whole roadmap.
Then there’s the benchmark evidence, which is less flattering to custom pipelines than most of us expected. Stanford’s RAGGED study, published in October 2024, evaluated more than 40 retrieval and RAG configurations across 43 datasets. The headline finding: no single architecture dominates. Fine-tuning didn’t reliably beat prompt-based methods either. If no configuration reliably wins, the rational default is the simplest thing that works. For a lot of use cases, that’s now the provider’s file search, not the eleven-stage RAG stack you’ve been maintaining.
Maintenance tilts the same direction. Every embedding model deprecation you don’t manage is a weekend you get back. The provider handles index rebuilds, scaling, and the retrieval internals when models change.
Citations help with the hallucination problem, the one buyers keep naming. The best models on Vectara’s Hallucination Leaderboard still invent roughly 1 to 2 percent of sentences in summarization tasks. Grounded answers with visible sources don’t eliminate that, but they let users check the source, which is most of the battle for trust.
The fit is clearest for prototypes, internal knowledge tools with document counts in the low thousands, web-factual product features, and demo environments. Gartner predicted in 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025. Cheap, grounded retrieval is exactly how you avoid joining that statistic during the evaluation phase.
Where It Breaks at Enterprise Scale
Then the part provider launch posts skip. Four failure modes show up reliably once real constraints arrive.
Permissions
Provider file search retrieves whatever sits in the store you gave it. It has no idea that user A can see contracts X and Y but not Z, because that logic lives in your identity system, not theirs. Document-level sharing is the best most providers offer. Row-level, record-level, and role-based filtering before retrieval is your RAG stack’s job. OWASP’s Top 10 for RAG applications lists access control and sensitive information disclosure among its top risks, and shared provider storage is a common source. If your answers must respect per-user access, native file search is disqualified for that corpus. Not a hard call.
Compliance and residency
Documents in provider storage have left your VPC. Retention windows, legal holds, residency requirements, and audit trails all get harder to argue. If you operate under the EU AI Act or handle regulated data, “which provider region stored this chunk” becomes a real compliance question. Your own index gives you exact answers. A managed store gives you a policy document.
Control, ranking, and evals
You can’t tune hybrid search weights you don’t own. You can’t inspect retrieval scores, A/B your chunking, or rerank with your own model. When a provider updates its retrieval internals, behavior shifts and you find out from users. LangChain’s State of AI Agents survey of more than 1,300 engineers named accuracy and hallucinations as the top two reasons agent projects struggle in production, and you can’t fix what you can’t measure. Debugging a bad grounded answer becomes a support ticket instead of a query log.
Unit economics
Per-query pricing is nearly free at small volume and rude at scale. Here’s a template, not a quote. If native search costs about $2.50 per 1,000 queries and your fixed RAG stack runs $800 a month, the crossover sits near 320,000 queries a month. Most early teams never get there. A product with real traffic gets there in a bad week. Run the math with your actual rates, then re-run it every quarter, because both sides keep moving.
5 Decisions to Make Before You Touch Your RAG Stack
If the trend is real, the response isn’t panic or denial. It’s drawing the boundary on purpose. These five decisions get you there.
-
Split your corpus by trust level. Public or low-stakes content can ride in provider retrieval. Proprietary corpora stay in your index. Write both lists down. The act of writing surfaces the edge cases you were about to miss.
-
Name an owner for permissions. If answers must respect per-user access, that data cannot live in a shared provider store. The index that enforces ACLs keeps that workload. Full stop.
-
Draw the compliance line with legal in the room. Residency, retention, and audit requirements decide where documents physically live. Engineering proposes, legal signs. If legal won’t sign the boundary, the boundary moves.
-
Run the break-even math. Pull last month’s query volume, multiply by provider rates, compare against your index’s fixed costs, and note the crossover. Put the number where your team can see it. When query volume doubles after a launch, you’ll already know what that does to the invoice.
-
Build the router before you need it. A cheap classifier, or the model itself, can send simple queries to native retrieval and hard ones to your pipeline. Keep a golden set of 100 real questions and eval both paths monthly. When one path degrades, you’ll know which one and why.
One pattern is already common in production: native retrieval handles the long tail of web questions, your RAG stack serves the core product surface, and the router decides between them. The router lives in your code, which keeps the boundary yours to move as prices and capabilities shift.
Where This Leaves Your RAG Stack
Retrieval isn’t dying. The boundary between the provider’s layer and yours is moving, and it will keep moving toward the provider for simple use cases. Your RAG stack earns its keep where permissions, compliance, control, and volume demand it, and nowhere else.
So the next time a PM asks why you still run a vector database, skip “because we built it.” Try this instead: a corpus split by trust level, a named owner for permissions, a compliance line legal actually signed, a crossover number on the wall, and monthly evals proving the split still works. That’s not a defensive answer. It’s an architecture.
We cover calls like these every week at Rag About It. Subscribe for teardowns of production RAG setups, benchmark digests, and the retrieval decisions that never make the launch posts. And if you’re redrawing your RAG stack and need documentation that keeps pace with the new boundary, that’s what we do. Start with the guides, and talk to us when you’re ready.
