AI Agents for Beginners - 5. Agentic RAG

Agentic RAG — how LLMs autonomously decide when and how to retrieve information, iterate with a maker-checker loop, self-correct, and deliver high-quality grounded responses.

June 8, 2026

ai ai-agents

AI Agents for Beginners - 5. Agentic RAG

This post summarizes key takeaways from Lesson 05 of Microsoft's AI Agents for Beginners course.

Source: microsoft/ai-agents-for-beginners - 05-agentic-rag

What is Agentic RAG?

Agentic RAG (Retrieval-Augmented Generation) is a new AI paradigm where LLMs retrieve information from external sources while autonomously planning their next steps.

Key differences from traditional RAG:

	Traditional RAG	Agentic RAG
Retrieval Method	Fixed pipeline (retrieve → generate)	LLM autonomously decides when and how to search
Iterativity	Generates response after a single search	Evaluates results and repeats retrieval if needed
Query Handling	Static query	Rewrites query or changes approach upon failure
Orchestration	Requires complex prompt chains	Simple LLM call → tool use loop

This iterative Maker-Checker approach enhances accuracy, handles flawed queries, and ensures high-quality results.

The Core Loop

At the heart of Agentic RAG is an iterative loop of LLM call → tool use → LLM call → ….


Agentic RAG Core Loop
Mermaid
flowchart TD
    Start["User Goal (Prompt)"]
    --> LLM["LLM\nAnalyze Goal & Plan"]
    LLM --> Decision{Information\nSufficient?}
    Decision -->|"No"| Tool["Invoke Tool / Retrieval\n(Vector Search, SQL, API)"]
    Tool --> Assess["Assess Results\n& Update Memory"]
    Assess --> Refine{Need Query\nRefinement?}
    Refine -->|"Yes"| Tool
    Refine -->|"No"| Decision
    Decision -->|"Yes"| Response["Generate Final Response"]

Steps in the loop:

Step	Description
Initial Call	The user goal (prompt) is passed to the LLM
Tool Invocation	If information is insufficient, an appropriate tool such as vector search, SQL, or an API is selected and invoked
Assessment & Refinement	Evaluates returned data; if insufficient, refines the query or tries another tool
Repeat Until Satisfied	Repeats the cycle until sufficient information is gathered
Memory & State	Remembers results from each step to avoid redundant loops and make better decisions

Owning the Reasoning Process

A defining characteristic of an agentic system is that it owns its reasoning process.
While traditional RAG follows human pre-defined paths, Agentic RAG autonomously decides the sequence of steps based on the quality of information.

For instance, when tasked with creating a product launch strategy, the agent autonomously decides to:

Search for current market trend reports using Bing Web Grounding
Identify competitor data using Azure AI Search
Correlate past internal sales metrics using Azure SQL Database
Synthesize everything into a coherent strategy using Azure OpenAI Service
Evaluate gaps and inconsistencies in the strategy, repeating retrieval if necessary

All these steps are decided by the model itself, without human pre-scripting.

Building a RAG Agent

Creating a Search Tool

By wrapping external data sources into tools, the agent can invoke retrieval whenever needed:


search_tool.py
python

Basic RAG Agent

Instruct the agent to always search first, generating responses grounded in the knowledge base rather than relying solely on its training data:


rag_agent.py
python

Maker-Checker Pattern

Iterative retrieval is a core strength of Agentic RAG.
The agent performs multiple searches to verify and supplement initial results:

Maker step — Retrieves initial information and drafts a response
Checker step — Performs additional searches to confirm details or fill in gaps


maker_checker_agent.py
python


Maker-Checker Pattern Flow
Mermaid
zenuml
    title Maker-Checker Pattern Flow
    User->Agent: budget $175/day, travel in April
    Agent->Tool: search_travel_knowledge(April travel)
    Tool->Agent: initial destination results
    Agent->Agent: evaluate completeness
    Agent->Tool: search_travel_knowledge(Barcelona)
    Tool->Agent: detailed info with cost and season
    Agent->Tool: search_travel_knowledge(Paris)
    Tool->Agent: detailed info with cost and season
    Agent->Agent: compare and verify against budget
    Agent->User: verified recommendation with details

Handling Failure Modes and Self-Correction

The autonomy of Agentic RAG includes robust self-correction mechanisms.
Even when irrelevant documents are retrieved or queries fail, the system recovers on its own:

Failure Mode	Self-Correction Method
Irrelevant Results	Try a new search strategy, rewrite the query, or explore alternative datasets
Flawed Queries	Debug reasoning steps with diagnostic tools and verify retrieved data accuracy
Repeated Failures	Flag uncertainty and request human review. Learn from feedback

Through this iterative and dynamic approach, the agent goes beyond a simple one-shot system to learn from mistakes within a session and continuously improve.

Boundaries of Agency

While Agentic RAG is powerful, it is distinct from Artificial General Intelligence.
Its capabilities as an "agent" are bounded by the tools, data sources, and policies provided by human developers:

Boundary	Description
Domain-Specific Autonomy	Operates to achieve user-defined goals within a known domain. Improves outcomes via query rewriting and tool selection
Infrastructure-Dependent	Limited strictly to the scope of tools and data integrated by developers. Cannot transcend these boundaries without human intervention
Respect for Guardrails	Ethical guidelines, compliance rules, and business policies are maintained at all times

Practical Use Cases

Use Case	Description
Correctness-First Environments	In compliance reviews, regulatory analysis, and legal research, iteratively verifies multiple sources to produce thoroughly validated answers
Complex Database Interactions	Autonomously refines failed queries when processing structured data with Azure SQL or Microsoft Fabric
Extended Workflows	Continuously adapts strategies across long-running sessions as new information emerges

Governance, Transparency, and Trust

As agents reason more autonomously, governance and transparency become increasingly vital:

Explainable Reasoning — Provide audit trails of queries executed, sources referenced, and reasoning steps taken. Maintain transparency with Azure AI Content Safety and Azure AI Tracing
Bias Control — Calibrate retrieval strategies to utilize balanced, representative data sources, and regularly audit for bias and distortion patterns
Human Oversight — Human review is indispensable for sensitive tasks. Agentic RAG does not replace human judgment, but rather complements it by presenting thoroughly vetted options

Summary


Lesson 05 Summary
Mermaid
flowchart LR
    Root["Agentic RAG"]
    Root --> Core["Core Loop"]
    Root --> Pattern["Maker-Checker"]
    Root --> Self["Self-Correction"]
    Root --> Gov["Governance"]
    Core --> C1["LLM call → Tool → Evaluate"]
    Core --> C2["Maintain Memory & State"]
    Core --> C3["Own Reasoning Process"]
    Pattern --> P1["Maker: Initial Search & Draft"]
    Pattern --> P2["Checker: Re-search & Verify"]
    Self --> S1["Query Rewriting"]
    Self --> S2["Try Alternative Tools"]
    Self --> S3["Human Oversight Fallback"]
    Gov --> G1["Explainable Reasoning"]
    Gov --> G2["Bias Control"]

Agentic RAG moves beyond the fixed pipeline of traditional RAG, allowing the LLM to autonomously decide retrieval strategies
The Maker-Checker loop iteratively verifies initial findings to enhance accuracy
In case of failure, robust self-correction is enabled via query rewriting, tool substitution, and human oversight fallbacks
As autonomy increases, trustworthy governance must be upheld through Explainable Reasoning, Bias Control, and Human Oversight

PreviousAI Agents for Beginners - 6. Building Trustworthy AI Agents

NextAI Agents for Beginners - 4. Tool Use Design Pattern