Cua

What is this?

Cua ("Computer-Use Agent") gives AI agents computers they can use: open-source desktop automation drivers, isolated cloud desktops, local macOS VMs, specialist decision models, and benchmarks for evaluating computer-use agents.

With 26k+ stars and an MIT license, Cua is one of the largest open-source platforms for Computer-Use 2.0 — an agent moving between code, APIs, and graphical interfaces within the same task, instead of being stuck in a chat window.

What problem does it solve?

Building an agent that can actually use a computer is brutally hard:

Text-only agent:   You → Prompt → Text
                   (can't click, can't see, can't verify)

Computer-use:      Agent → sees screen → moves cursor → clicks
                   → reads result → decides next step
                   → repeats, safely, on real UIs

To do that you need things that rarely exist in one place:

• A way to inspect and drive native apps on macOS, Windows, and Linux
• Isolated machines to run on (so a rogue agent can't wreck your laptop)
• Local VMs for privacy-sensitive or offline work
• Specialist models for fast, bounded UI decisions ("which value goes in this field?")
• Benchmarks to know if any of it actually works

Cua ships all five as one open-source platform.

How does it work?

                ┌─────────────────────────────────┐
                │        Your agent / model       │
                │  (Claude Code, Codex, Cursor…)  │
                └──────────────┬──────────────────┘
                               │ CLI · MCP · typed SDK
                ┌──────────────▼──────────────────┐
                │          Cua Driver             │
                │  inspect & operate apps +       │
                │  browsers on macOS/Win/Linux    │
                └──────┬───────────────┬──────────┘
                       │               │
      ┌────────────────▼──┐   ┌────────▼─────────────┐
      │  Cua Fleets       │   │  Lume                │
      │  isolated cloud   │   │  local macOS/Linux   │
      │  desktops (pool)  │   │  VMs on Apple Silicon│
      └───────────────────┘   └──────────────────────┘
                       │
      ┌────────────────▼──────────────────────────┐
      │  CUA-S1 (fast System 1 decisions)         │
      │  Cua Bench (tasks · eval · trajectories)  │
      └───────────────────────────────────────────┘

1. Your agent asks the Cua Driver for a screenshot or to perform an action.
2. The driver runs inside a Fleet cloud desktop or a Lume local VM — never loose on your machine unless you want that.
3. For tight UI decisions, CUA-S1 specialist models score options fast; the application code orders actions and the driver executes with explicit boundaries.
4. Cua Bench gives you tasks with evaluators so you can measure whether the whole loop actually works — and export trajectories for training.

Repository Structure

trycua/cua/
├── libs/
│   ├── cua-driver/       # Desktop automation driver (macOS/Win/Linux)
│   ├── cua-bench/        # Task creation, evaluation, trajectories
│   ├── cua-bench-s1/     # Bench variant for S1 decisions
│   ├── cua-s1/           # Specialist System 1 model code + training
│   ├── lume/             # Local macOS/Linux VMs on Apple Silicon
│   ├── lumier/           # Lume companion tooling
│   ├── fleet/            # Cloud fleet orchestration (Cua Fleets)
│   ├── kasm/             # Containerized desktops (third-party, MIT)
│   ├── python/           # Python SDKs
│   └── typescript/       # TypeScript SDKs
├── samples/              # driver/ and python/ examples
├── skills/               # gui-automation, jev-use agent skills
├── docs/                 # Full documentation site (cua.ai/docs)
├── tests/                # integration + agent-loop tests
├── infra/ · clusters/    # Cloud infrastructure definitions
├── rfcs/                 # Design proposals
└── pyproject.toml · uv.lock · package.json

Key Features

🖥️ Cua Driver — Give your agent tools to inspect and operate native desktop apps and browsers on macOS, Windows, and Linux. Connect through the CLI, MCP, or typed SDKs. Background delivery lets agents work without moving your pointer or taking focus when the platform supports it.

☁️ Cua Fleets — Provision isolated cloud desktops at run.cua.ai. A Fleet maintains sandbox capacity; your code claims a desktop from a pool and uses the Sandbox SDK to run commands, capture screenshots, and interact with apps inside it.

🍎 Lume — Create and manage local macOS and Linux VMs on Apple Silicon using Apple's Virtualization.Framework. One curl command installs it; build a macOS Tahoe VM from an Apple restore image and connect over SSH.

⚡ CUA-S1 — A family of small, specialized "System 1" models for fast, bounded computer-use decisions (choosing which value belongs in a field, whether to leave an element alone). Includes Python model code, synthetic-data generation, training, and evaluation — weights hosted on Hugging Face.

📊 Cua Bench — Build computer-use tasks, evaluate agents, and export trajectories for training. Start with a simulated task that needs no VM, Docker, or model API key — your first result is a task whose evaluator reports reward 1.0.

🔐 Bring Your Own Agent — Works with Claude Code, Codex, Cursor, OpenClaw, or any MCP-capable agent. Cua provides the computer and automation tools; you keep your model and your workflow.

What You'll Learn

• How computer-use agents actually drive real desktop UIs (screenshots → decisions → actions)
• How to isolate agent execution in cloud fleets and local VMs
• How to build a cross-OS automation driver with CLI, MCP, and SDK interfaces
• How to evaluate agent behavior with reproducible tasks and trajectory exports
• How specialist small models handle bounded UI decisions alongside a larger agent
• How to run macOS virtualization on Apple Silicon with Virtualization.Framework