AI Agents for Beginners - 6. Building Trustworthy AI Agents

How to build safe and trustworthy AI agents — system message framework, threat mitigation, and human-in-the-loop patterns for secure, reliable agentic systems.

June 9, 2026

ai ai-agents

AI Agents for Beginners - 6. Building Trustworthy AI Agents

This article summarizes Lesson 06 of Microsoft's AI Agents for Beginners course.

Source: microsoft/ai-agents-for-beginners - 06-building-trustworthy-agents

Safety: System Message Framework

The first key element in building a secure agent application is Safety.
It means ensuring that the agent behaves exactly as designed.

While system prompts are always critical in LLM-based applications, they are even more crucial for AI Agents.
An agent requires highly specific instructions to correctly carry out its intended tasks.

The System Message Framework is an effective method for generating scalable system prompts:


System Message Framework
Mermaid
flowchart TD
    Step1["Step 1\nWrite Meta System Message\n(Instruct LLM on prompt generation)"]
    --> Step2["Step 2\nWrite Basic Prompt\n(Describe agent's role & tasks)"]
    --> Step3["Step 3\nOptimize with LLM\n(Meta + Basic → Final System Message)"]
    --> Step4["Step 4\nIterate & Improve\n(Small edits → Repeat evaluation & comparison)"]

Step 1: Meta System Message

This is a meta-prompt that guides the LLM on how to generate the system prompt:


meta_system_message.txt
python

Step 2: Basic Prompt

Concisely describe the agent's role and tasks:


basic_prompt.txt
python

Step 3: Optimize with LLM

Passing the Meta System Message as the system prompt and the Basic Prompt as the user prompt generates a structured system prompt:


generate_system_message.py
python

The core of Step 4 is iteration and improvement.
Gradually optimize by making slight modifications to the Basic Prompt and comparing the results.

Understanding Threats

To build trustworthy AI agents, you must understand and mitigate potential threats.

Threat	Description	Mitigation
Task & Instruction Manipulation	Attackers manipulate prompts or inputs to alter the agent's instructions or goals	Input validation and filtering, limiting conversation turns
Access to Critical Systems	If the agent accesses sensitive systems, attackers may hijack communications or extract system information	Least privilege (need-only) access, authentication/access control, secure communication
Resource & Service Overloading	Exploiting the agent's tool access with high request volumes, causing service outages or excessive costs	Rate limiting policies, capping conversation turns and request counts
Knowledge Base Poisoning	Corrupting the agent's knowledge base to induce biased or unintended responses	Regular data validation, access control allowing only trusted personnel to modify data
Cascading Errors	An error in one connected system cascades into other systems	Running in isolated environments such as Docker containers, implementing fallback and retry logic upon errors


Threat Categories & Mitigations
Mermaid
flowchart LR
    Threats["AI Agent\nThreats"]
    Threats --> T1["Task & Instruction\nManipulation"]
    Threats --> T2["Critical System\nAccess"]
    Threats --> T3["Resource\nOverloading"]
    Threats --> T4["Knowledge Base\nPoisoning"]
    Threats --> T5["Cascading\nErrors"]
    T1 -->|"Mitigation"| M1["Input validation\n+ Turn limits"]
    T2 -->|"Mitigation"| M2["Least privilege\n+ Auth/ACL"]
    T3 -->|"Mitigation"| M3["Rate limiting\n+ Request caps"]
    T4 -->|"Mitigation"| M4["Data verification\n+ Trusted access"]
    T5 -->|"Mitigation"| M5["Isolation\n+ Fallback logic"]

Human-in-the-Loop

Another key approach to building trustworthy AI agent systems is Human-in-the-Loop.
It establishes a flow where users can provide feedback during agent execution and decide whether to approve or abort.

The user functions like another agent in a multi-agent system, supervising the execution process.


human_in_the_loop.py
python


Human-in-the-Loop Flow
Mermaid
zenuml
    title Human-in-the-Loop Flow
    User->Agent: task request
    Agent->LLM: process request
    LLM->Agent: draft response
    Agent->Human: present draft for review
    Human->Agent: APPROVE or REJECT
    Agent->User: deliver approved response

When used alongside the approval_mode="always_require" tool, human approval can be enforced for every sensitive operation.

Summary


Lesson 06 Summary
Mermaid
flowchart LR
    Root["Trustworthy\nAI Agents"]
    Root --> Safety["Safety"]
    Root --> Threats["Threats"]
    Root --> HITL["Human-in-the-Loop"]
    Safety --> S1["Meta System Message"]
    Safety --> S2["Basic Prompt"]
    Safety --> S3["Iterate & Improve"]
    Threats --> T1["Prompt Injection"]
    Threats --> T2["System Access"]
    Threats --> T3["Resource Overload"]
    Threats --> T4["KB Poisoning"]
    Threats --> T5["Cascading Errors"]
    HITL --> H1["User approval &\nabort permissions"]
    HITL --> H2["approval_mode\nalways_require"]

Design scalable and safe prompts using the 4 steps of the System Message Framework (Meta → Basic → Optimize → Iterate).
Understand the five major threats—including Prompt Injection, Knowledge Base Poisoning, and Cascading Errors—and apply appropriate mitigation strategies for each.
Supervise agent execution with Human-in-the-Loop, requiring mandatory approval for sensitive actions with approval_mode="always_require".
Continuously iterating and improving Safety, Security, and Privacy is at the heart of building trustworthy AI Agents.

NextAI Agents for Beginners - 5. Agentic RAG