AI Agents for Beginners - 15. Computer Use Agent (CUA)

Build a Computer Use Agent (CUA) with Browser-Use and Playwright via CDP integration. Covers vision-based navigation, structured Pydantic extraction, Agent vs Actor pattern selection, and a real-world Airbnb price comparison workflow.

June 9, 2026

ai ai-agents

AI Agents for Beginners - 15. Computer Use Agent (CUA)

This article summarizes Lesson 15 of Microsoft's AI Agents for Beginners course.

Source: microsoft/ai-agents-for-beginners - 15-browser-use

A Computer Use Agent (CUA) is an agent that interacts with websites just like a human by opening a browser, viewing pages, and taking the best actions on the screen.
In this lesson, we will build a browser automation agent that searches Airbnb to find the cheapest accommodation in Stockholm.

To achieve this, we combine four key technologies:

Browser-Use: AI-powered browser navigation
Playwright + CDP: Browser lifecycle control
Azure OpenAI (GPT-4 Vision): Vision-based reasoning
Pydantic: Structured data extraction

Architecture Overview

The notebook demonstrates a hybrid browser automation workflow:

Chrome is launched with CDP (Chrome DevTools Protocol) enabled so that Playwright and Browser-Use can share the same browser session.
The Browser-Use agent handles open-ended navigation tasks, such as opening Airbnb, dismissing pop-ups, and searching for Stockholm.
The active page is inspected using a structured Pydantic schema to extract the listing title, price per night, rating, and URL.
Python logic compares the extracted listings and highlights the cheapest result.


CDP Architecture
Mermaid
%%{init: {'look': 'handDrawn'}}%%
flowchart TD
    subgraph Chrome["Chrome Browser (CDP Port 9222)"]
        Page["Active Page"]
    end
    PW["Playwright\nManage browser lifecycle\nHandle CDP connection"] -->|"connect_over_cdp"| Chrome
    BU["Browser-Use\nAI-powered automation\nExtract structured output"] -->|"cdp_url"| Chrome
    Agent["Agent\n(use_vision=True)"] --> BU
    LLM["Azure OpenAI\nGPT-4 Vision\nScreenshot analysis"] --> Agent
    Pydantic["Pydantic Models\nAirbnbListing\nSearchResult"] --> BU

Playwright vs Browser-Use

Role	Playwright	Browser-Use
Browser Lifecycle Management	✅ Robust control	—
CDP Connection	✅ Full support	✅ via cdp_url
AI-based Element Discovery	❌ Requires CSS selectors	✅ Uses natural language
Vision-based Page Understanding	❌	✅ Screenshot analysis
Structured Output Extraction	❌	✅ Pydantic integration
Timing & Wait Control	✅ Complete control	△ Less predictable

Setup


setup.sh
bash

Set up the Azure OpenAI environment variables:


.env
bash

Azure OpenAI LLM Initialization

Initialize the LLM using Browser-Use's ChatAzureOpenAI. Set temperature to 0.3 to achieve consistent and predictable automation:


airbnb_agent.py
python

Structured Output Models

Define a type-safe schema for data extraction by the agent using Pydantic models:


airbnb_agent.py
python

Starting Chrome with CDP

Launch Chrome with an open CDP port so that Playwright and Browser-Use can share the same Chrome instance:


airbnb_agent.py
python


Starting Chrome with CDP
Mermaid
%%{init: {'look': 'handDrawn'}}%%
sequenceDiagram
    participant Script
    participant Chrome
    participant Playwright
    participant BrowserUse

    Script->>Chrome: Launch (--remote-debugging-port=9222)
    Script->>Playwright: connect_over_cdp("http://localhost:9222")
    Playwright-->>Script: playwright_browser
    Script->>BrowserUse: Browser(cdp_url="http://localhost:9222", keep_alive=True)
    Note over Playwright,BrowserUse: Share the same Chrome session
    Script->>BrowserUse: Agent(task=..., browser=browser).run()
    BrowserUse->>Chrome: Browser automation (AI-based)
    Playwright->>Chrome: Direct page control (deterministic)

AirbnbSearchAgent Class

A search agent that strategically combines the Agent and Actor approaches:


airbnb_agent.py
python


Agent Search Workflow
Mermaid
%%{init: {'look': 'handDrawn'}}%%
flowchart LR
    subgraph Step1["Step 1: Agent Navigation (AI)"]
        A1["Open https://airbnb.com"] --> A2["Dismiss pop-ups & cookie banners"]
        A2 --> A3["Search 'Stockholm, Sweden'"]
        A3 --> A4["Verify results page loaded"]
    end
    subgraph Step2["Step 2: Vision Extraction"]
        E1["page.extract_content()"] --> E2["GPT-4 Vision analyzes page screenshot"]
        E2 --> E3["Extract all listing prices"]
        E3 --> E4["Return as SearchResult Pydantic model"]
    end
    Step1 --> Step2
    E4 --> R["Cheapest listing\nAverage price\nPrice range"]

Running the Full Workflow


airbnb_agent.py
python

Example execution output:


Output
bash

When to Use Agent vs Actor

A key design decision in CUA implementation is when to use an Agent vs. when to use an Actor (direct control):

Scenario	Use Agent	Use Actor
Dynamic Layouts	✅ AI adapts to page changes	❌ CSS selectors break
Known Structure	❌ Slower than direct control	✅ Fast and accurate
Finding Elements	✅ Uses natural language queries	❌ Requires exact selectors
Timing Control	❌ Lower predictability	✅ Full control over waits & retries
Complex Workflows	✅ Handles unexpected UI states	❌ Requires explicit branching logic


Agent vs Actor
Mermaid
%%{init: {'look': 'handDrawn'}}%%
flowchart LR
    Task["New Task"] --> Q1{"Is the layout\ndynamic?"}
    Q1 -->|"Yes"| Q2{"Is structured\nextraction needed?"}
    Q1 -->|"No"| Actor["Actor Pattern\nCSS Selectors\n+ Direct Playwright Control"]
    Q2 -->|"Yes"| Hybrid["Hybrid\nAgent Navigation +\npage.extract_content()"]
    Q2 -->|"No"| Agent["Agent Pattern\nuse_vision=True\n+ Natural Language Task"]

Browser-Use Best Practices

Start with an agent for exploration and dynamic navigation — AI automatically handles pop-ups, cookie banners, and layout changes
Switch to direct page control when the interaction becomes predictable — Actors are faster and more reliable for repetitive tasks
Use structured output models — Ensure type safety and validation of extracted data using Pydantic
Add delays after visible UI changes — Allow sufficient wait time after clicking search buttons or page transitions
Capture screenshots for debugging — Capture visual states during agent execution with page.screenshot()
Design fallback strategies for pop-ups and layout shifts — Websites can change at any time
Blend agent and actor patterns — A hybrid approach to achieve both flexibility and precision

Real-World Applications

The Browser-Use + Playwright CUA pattern can be applied to a variety of real-world scenarios:

Travel Booking: Price monitoring, automated bookings, option comparisons
E-commerce: Inventory tracking, price comparison, automated purchasing
Structured Data Extraction: Type-safe data scraping from dynamic websites
Vision-aware UI Testing: Automated testing where AI visually inspects and verifies screens like a human
Website Monitoring: Change detection, alerts for specific conditions
Intelligent Form Filling: Intelligently filling out complex forms with AI


CUA Applications
Mermaid
%%{init: {'look': 'handDrawn'}}%%
flowchart TD
    CUA["Computer Use Agent\n(Browser-Use + Playwright + CDP)"]

    CUA --> Nav["AI Navigation\nuseVision=True\nNatural Language Tasks"]
    CUA --> Extract["Structured Extraction\npage.extract_content()\nPydantic Models"]
    CUA --> Control["Direct Control\nPlaywright Actor\nDeterministic Behavior"]

    Nav --> UC1["Dynamic Websites\nPop-up & Banner Handling"]
    Extract --> UC2["Price Comparison\nData Collection"]
    Control --> UC3["Form Automation\nUI Testing"]

    UC1 --> Result["Real-World Use Cases\nTravel Booking · E-commerce\nMonitoring · Testing"]
    UC2 --> Result
    UC3 --> Result

NextAI Agents for Beginners - 14. Microsoft Agent Framework