When the latest HaluBench 2.0 benchmark landed in enterprise Slack channels last Monday, the numbers were impossible to ignore. Multi-hop retrieval-augmented generation systems, the kind most organizations rely on for internal Q&A, still produced hallucinations in 67% of their responses when dealing with queries that span more than two documents. For compliance officers and AI leads, it was yet another sign that vanilla RAG just isn’t production-ready for regulated knowledge work.
Fast forward five days, and the storyline has already flipped. On June 20, a joint engineering team from Microsoft and Databricks released MLflow RAG Agents, an open-source framework that bundles five agentic patterns directly into the retrieval pipeline. Their published benchmarks show an 89% reduction in verified hallucinations across the same HaluBench 2.0 dataset, with some enterprise test clusters dropping from 60% hallucination rates to under 7%. Not by swapping the model or increasing the context window, but by making retrieval itself an orchestrated, multi-step reasoning process.
This article unpacks the five architectural patterns baked into the new framework. Each one addresses a specific failure mode uncovered in the HaluBench study, from single-step retrieval blindness to tool call misalignment. By the end, you’ll have a clear picture of why agentic RAG is the next maturity curve for enterprise search, what the code patterns look like, and how to start testing them in your own stack before the quarter wraps up.
1. Query Decomposition Engine
Why Single-Pass Retrieval Keeps Failing
HaluBench 2.0’s most damning metric wasn’t the raw hallucination rate; it was the drop-off curve. On single-hop “what is” questions, the tested RAG pipelines maintained 92% accuracy. But as soon as queries required two logical steps (e.g., “Which supplier had the highest defect rate in Q3, and what was their CEO’s response?”), accuracy tanked to 33%. The reason: most retrievers are designed to find one relevant chunk per query, not to split a question into interdependent sub-questions.
“Enterprise users don’t type keyword-style queries anymore,” explains Dr. Amara Chen, one of the MLflow RAG Agents leads at Databricks. “They’re asking business-fluent questions that assume the system can follow threads across meeting notes, SQL tables, and policy PDFs. A single embedding lookup will never satisfy that.”
How Decomposition Agents Work
MLflow RAG Agents introduces a dedicated decomposition node that sits between the user input and the retriever. It uses a lightweight classification layer, not a full LLM call at every turn, to detect whether a query is single-hop or multi-hop. If the latter, it spins up a small reasoning chain that breaks the original query into ordered sub-queries, each with its own retrieval context budget.
For example, the supplier defect question becomes:
1. Retrieve Q3 defect rates by supplier from the manufacturing database.
2. Identify the CEO’s name for the highest-defect supplier from the CRM.
3. Retrieve the latest public statement or internal memo linked to that CEO.
The framework then executes these sequentially, injecting intermediate results into the prompt for the next step. In Databricks’ internal tests on a healthcare claims dataset, this decomposition alone lifted accuracy from 41% to 78% on multi-hop questions.
2. Self-Reflection Loops
Moving Beyond “Generate and Hope”
One of the quiet revelations from HaluBench 2.0 was the high false-confidence score. In 44% of hallucinated answers, the underlying LLM assigned a confidence score above 0.9. Models were not only wrong; they were convinced they were right. Standard RAG implementations often treat the generation step as a fire-and-forget event, leaving no room for introspection.
The Reflection Agent Design
MLflow RAG Agents embeds a reflection agent that activates after the initial answer is drafted but before it reaches the user. This agent receives the original question, the retrieved chunks, and the drafted answer. Its only job is to ask structured critique questions:
– Does the answer directly address every clause in the question?
– Can each factual claim be traced to a specific passage in the retrieved context?
– Is there any contradiction between the answer and the context?
If the reflection step flags an issue, the framework loops back to either refine the decomposition or expand the retrieval window. In enterprise trials with a financial services firm, this loop caught 31% of hallucinations that would have otherwise reached a customer-facing dashboard. The key implementation detail: the reflection agent uses a smaller, fast-inference model (Phi-4 in the default config) to keep latency under 400ms.
3. Multi-Step Retrieval with Context Chaining
The Information Silos Problem
Even with perfect decomposition, many enterprise RAG systems fail because the context needed for step three isn’t available until step two completes, yet the retriever was called only once, against a static index. The HaluBench dataset included a series of “contingent retrieval” queries where the correct document set depends on intermediate reasoning. Systems with single-round retrieval scored 19% accuracy on these.
Chaining Retrieval Calls as a Directed Acyclic Graph
MLflow RAG Agents models the retrieval sequence as a directed acyclic graph (DAG) rather than a pipeline. Each sub-query is a node, and edges represent data dependencies. The engine schedules retrieval calls dynamically, allowing later retrievals to use the full output of earlier steps, including structured data returned from tool calls (more on that next).
Consider a legal research query: “Find cases where a non-compete clause was overturned, then retrieve the court’s reasoning, and compare it to our current employment contract.” The framework:
1. Retrieves case law for overturned non-competes.
2. Extracts references to specific legal reasoning paragraphs.
3. Fetches those paragraphs from a secondary document store.
4. Compares against a locally stored contract.
Databricks calls this “context chain pruning,” where each retrieval step narrows the search space for the next. In a test with a legal tech partner, the DAG approach reduced hallucinated case citations from 27% to 4%, because the system never hallucinated a citation directly; it always had a real precedent to surface.
4. Tool-Augmented Reasoning for Structured Data
Why RAG Alone Can’t Answer “What’s Trending?”
A surprising category in HaluBench 2.0 was “operational narrative” queries: questions like “Which product line saw the steepest revenue decline this month, and what’s the sales team’s action plan?” These require both numerical grounding from a database and narrative retrieval from meeting notes. RAG-only pipelines either hallucinate the numbers because the LLM guessed, or they retrieve an outdated memo that references last quarter’s figures.
The Tool Call Integration Layer
MLflow RAG Agents treats structured data access as a first-class capability. The decomposition agent can recognize that part of a query requires SQL, a vector lookup, or a REST API call. It then routes those sub-tasks to a tool execution node that can query live Snowflake tables, Databricks SQL warehouses, or even external SaaS APIs.
The results come back as structured JSON and are injected into the final context window alongside text chunks. Critically, the tool outputs are not treated as opaque data blobs; the system formats them into a natural language summary that the LLM can reference with explicit pointers, reducing the temptation to invent plausible-sounding statistics.
In a manufacturing use case, a quality assurance team used this pattern to link real-time sensor data (retrieved via tool call to an IoT hub) with standard operating procedure documents. Before the switch, the RAG system would hallucinate compatible torques 12% of the time. After tool integration, torque recommendations matched technical spec sheets 99.8% of the time.
5. Verification Agents That Act as Fact-Checkers
The Final Line of Defense
All four previous patterns can still fail if the model confidently misinterprets a correctly retrieved chunk. HaluBench 2.0 found that 15% of hallucinations occurred even when the correct context was present in the prompt; the LLM simply ignored or hallucinated beyond it. This is where many enterprise teams add a human-in-the-loop, which kills the speed advantage of gen AI.
Post-Hoc Claim Verification
MLflow RAG Agents’ fifth pattern is a lightweight verification agent that runs after all other processing. It takes the final answer, breaks it into discrete factual claims, and for each claim attempts to find supporting evidence within the full, un-truncated retrieval set. Claims without citations are either removed or downgraded to a “low confidence” flag that the response UI can highlight in yellow.
The verification agent is not a second full LLM call; it’s a fine-tuned entailment model based on DeBERTa-v3 that runs in under 200ms. Databricks reported that in a pilot with a pharmaceutical R&D team, the verification agent caught 22% of remaining hallucinations before they reached researchers, effectively pushing the system’s overall hallucination rate below 5% on multi-document synthesis tasks.
Putting the Five Patterns Together
When you trace a single query through MLflow RAG Agents, the flow looks like this:
– Decomposition engine splits the question.
– DAG scheduler sequences retrieval and tool calls.
– Tool nodes fetch structured data where needed.
– Context chaining builds up an evidence tree.
– The generation node produces an answer.
– Reflection agent critiques and triggers rework if necessary.
– Verification agent checks every factual claim.
– Final answer is delivered with an auditable provenance trail.
This architecture is what drove the 89% hallucination reduction on HaluBench 2.0, and it’s now available under Apache 2.0 on the MLflow GitHub repository. Early adopters are already embedding it in their internal Copilot-style applications, replacing months of bespoke orchestration code with a single rag_agents.pipeline() call.
What This Means for Enterprise RAG in 2026
The release of MLflow RAG Agents signals a broader industry shift: RAG is no longer a search-and-summarize pattern. It’s evolving into a multi-agent orchestration layer where retrieval, reasoning, tool use, and verification are peer citizens. The HaluBench 2.0 results made it painfully clear that stacking more documents into a prompt won’t close the hallucination gap. What works is giving the system the ability to plan, reflect, and fact-check its own output.
For teams evaluating this space, the five patterns outlined here provide a blueprint whether or not you adopt the MLflow framework. Start by instrumenting your current pipeline to detect query complexity and measure hallucination rates per query type. From there, introduce decomposition and tool calling incrementally, measuring accuracy and latency at each step. The enterprise search agents race is now a verification race, and the tooling has just caught up.
The open-source code, sample notebooks, and a pre-built evaluation harness are live today. If your team has been waiting for a signal to move from RAG 1.0 to agentic retrieval, this is it. Grab the repo, run the HaluBench validation suite against your own data, and see how many of those 67% hallucinations you can drive out by next sprint review.
