AI Agents for Beginners - 10. AI Agents in Production: Observability & Evaluation

AI Agents for Beginners - 10. AI Agents in Production: Observability & Evaluation

Source: microsoft/ai-agents-for-beginners - 10-ai-agents-production

Traces and Spans

Trace & Span Hierarchy
Mermaid
flowchart TD
    T["Trace\nEntire Agent Task\n(User Query Processing)"]
    T --> S1["Span: LLM Call\nModel Inference"]
    T --> S2["Span: Tool Call\nget_flight_info()"]
    T --> S3["Span: Tool Call\nget_activity_suggestions()"]
    T --> S4["Span: LLM Call\nFinal Response Generation"]
    S1 -->|"latency, tokens"| M1["Metrics"]
    S2 -->|"latency, result"| M2["Metrics"]
    S3 -->|"latency, result"| M3["Metrics"]
    S4 -->|"latency, tokens"| M4["Metrics"]

Why Observability Matters in Production Environments

Key Metrics to Track

Instrument your Agent

OpenTelemetry

otel_instrumentation.py
python

Manual Span Creation

manual_span_langfuse.py
python

Building an Observable Agent

setup.py
python

travel_tools.py
python

observable_agent.py
python

Observable Agent Flow
Mermaid
flowchart LR
    User["User Query\n'Plan a day trip in Paris'"]
    --> Timer["start_time = time.time()"]
    --> Agent["TravelAgent"]
    Agent --> F["get_flight_info('Paris')"]
    Agent --> A["get_activity_suggestions('Paris')"]
    F & A --> LLM["LLM\nSynthesize Response"]
    LLM --> Stop["elapsed = time.time() - start_time"]
    Stop --> Out["Response (elapsed s)\n+ Production: Send to OTel Backend"]

Agent Evaluation

Offline Evaluation

• Advantages: Repeatable with ground truth, providing clear accuracy metrics
• Key Challenges: Keeping test datasets comprehensive and continuously updated with real-world scenarios

Online Evaluation

• Advantages: Captures phenomena unforeseen in lab environments — model drift, unexpected query patterns
• Methods: Collecting implicit and explicit user feedback, shadow testing, A/B testing

Combining the two

Evaluation Loop
Mermaid
flowchart LR
    OE["Offline Evaluation\nEvaluate with Test Dataset"]
    --> Deploy["Deploy\nDeploy to Production"]
    --> OL["Online Monitoring\nTrack Real User Interactions"]
    --> Collect["Collect Failures\nGather New Failure Cases"]
    --> Add["Enrich Offline Dataset\nAdd Edge Cases & New Patterns"]
    --> Refine["Refine Agent\nImprove Prompts, Models, Logic"]
    Refine --> OE

Evaluation Patterns

evaluator_agent.py
python

Evaluator Agent Pattern
Mermaid
flowchart TD
    User["User Query"]
    --> Primary["Primary Agent\n(TravelAgent)"]
    Primary --> Resp["Agent Response"]
    Resp --> Eval["Evaluator Agent\n(ResponseEvaluator)"]
    Eval --> Score["Scores\nCompleteness · Accuracy · Helpfulness · Overall"]
    Score --> Gate{Meets Quality Criteria?}
    Gate -->|"Pass"| Deliver["Deliver to User"]
    Gate -->|"Fail"| Flag["Flag → Refine / Retry"]

Common Issues

Managing Costs

Cost Management: Router Model Strategy
Mermaid
flowchart TD
    Req["User Request"]
    --> Router["Router Model\n(LLM/SLM/serverless)"]
    Router --> Complexity{Classify Complexity}
    Complexity -->|"Simple\nIntent Classification & Parameter Extraction"| SLM["SLM\n(GPT-4o-mini, etc.)\nLow Cost · High Speed"]
    Complexity -->|"Complex\nComplex Reasoning & Planning"| LLM["Large Model\n(GPT-4o, etc.)\nHigh Performance"]
    Complexity -->|"Cached\nFrequently Asked Questions"| Cache["Cache\nInstant Response\nZero Cost"]
    SLM & LLM & Cache --> Response["Final Response"]

Lets see how this works in practice

Define Expense Models

expense_models.py
python

Defining Tools

expense_tools.py
python

Processing Expenses

expense_workflow.py
python

Expense Claim Workflow
Mermaid
flowchart LR
    Receipt["receipt.jpg\nReceipt Image"]
    --> OCR["OCRAgent\nload_receipt_image()\nImage → base64 Encoding"]
    OCR --> Parse["Parse Receipt Text\ndate|description|amount|category\nSemicolon-Separated Format"]
    Parse --> Email["EmailAgent\ngenerate_expense_email()\nExpenseFormatter.parse_expenses()"]
    Email --> Mail["Expense Claim Email\nDear Finance Team...\nTotal Amount: $xxx"]

Summary

Lesson 10 Summary
Mermaid
flowchart LR
    Root["AI Agents\nin Production"]
    Root --> Obs["Observability"]
    Root --> Eval["Evaluation"]
    Root --> Cost["Cost Management"]
    Obs --> O1["Traces & Spans"]
    Obs --> O2["Key Metrics\nLatency·Cost·Error·Feedback"]
    Obs --> O3["OpenTelemetry\n+ Manual Spans"]
    Eval --> E1["Offline Evaluation\nControlled Test Datasets"]
    Eval --> E2["Online Evaluation\nReal User Monitoring"]
    Eval --> E3["Evaluator Agent\nAutomated Quality Scoring"]
    Cost --> C1["Smaller Models (SLMs)"]
    Cost --> C2["Router Model"]
    Cost --> C3["Caching"]

• Turn agents from "black boxes" into "glass boxes" using Traces & Spans, enabling effective debugging and optimization.
• Track key metrics such as latency, cost, request errors, and user feedback to gain a comprehensive view of agent health.
• Establish pre-deployment baselines with Offline Evaluation, and capture real-world drift and unexpected patterns with Online Evaluation.
• Prevent regressions with the Evaluator Agent pattern by applying automated quality gating to primary agent responses.
• Manage costs using router models, SLMs, and caching while reserving high-performance models for tasks that require them.