Hermes Wiki
TechResearch/OpenWorker/openworker

OpenWorker — Architecture & Evaluation Notes

openworker.com · github.com/andrewyng/openworker (MIT)

From Andrew Ng's team. Positioned as a local-first, open-source "desktop AI agent" — the pitch is finished deliverables, not chat: "a polished document, a Slack reply with the numbers, an updated calendar, a triaged inbox." Also referred to internally as coworker (the Python package name).

1. What it actually is

A desktop app that wraps a local Python agent server. You point it at a workspace directory and it plans + executes multi-step tasks across your local files, terminal, and 25+ connected apps (Slack, Gmail, GitHub, Jira, Notion, HubSpot, etc.), plus anything reachable over MCP. Runs with your own API keys / local models — nothing routes through a vendor backend by default.

2. Architecture — three layers

  1. Desktop shell — Tauri (Rust) native shell supervising a React + Vite GUI. Also ships a Rust speech-to-text sidecar and a local secret store for credentials.
  2. Local Python agent server (coworker/) — the actual brain. Python 3.10+.
    • FastAPI app (coworker/server/app.py) is the control plane every surface (GUI, IDE, messaging integrations) rides on. It exposes:
      • An OpenAI-compatible chat completions endpoint
      • A WebSocket session API (carries the live agent engine event stream)
      • REST endpoints (settings, connectors, accounts, automations, subscriptions, etc.)
    • Served via uvicorn[standard].
    • pyproject.toml pins fastapi>=0.110, uvicorn[standard]>=0.27.
  3. Engine + tools + connectors, built on top of aisuite (a lightweight unified LLM API library, vendored/pinned rather than a loose PyPI dependency) for provider abstraction across OpenAI, Anthropic, Gemini, DeepSeek, Ollama (fully local), and Together/Fireworks for open-weight models.

So: FastAPI is the server framework for the local control-plane API, not a hosted backend — it's what the Tauri/React GUI (and any other "surface" — IDE plugin, messaging bot) talks to on localhost. This is a fairly clean example of "FastAPI as the glue layer between a native desktop shell and a Python agent engine" — relevant if we're evaluating patterns for Hermes_Agent or other local-first agent tooling.

3. Running from source

git clone https://github.com/andrewyng/openworker
bash packaging/setup_dev_env.sh
.venv/bin/openworker-server --cwd ~/some/project --port 8765

Prebuilt installers exist for macOS (Apple Silicon) and Windows x64 via openworker.com.

4. Key repo layout

  • coworker/ — Python backend (server, engine, providers, connectors, mcp client, permissions)
  • surfaces/gui/ — React/Tauri desktop interface
  • packaging/ — installers, dev env setup scripts
  • tests/ — extensive test suite, almost entirely built around fastapi.testclient.TestClient hitting coworker.server.create_app() — good signal that the REST/WS surface is the primary integration seam, tested black-box style.

5. Notable subsystems (from test file names, worth digging into further)

  • MCP client with OAuth support (coworker/mcp, test_mcp_oauth.py) — dynamic tool loading from arbitrary MCP servers, not just the bundled connectors.
  • Slack relay/gateway with approval-owner and allowlist concepts (test_slack_approval_owners.py, test_connectors_allowlist.py, test_team_allowlist.py) — suggests a permission-gating model for actions taken on connected accounts, not blind autonomy.
  • "Automations" and "subscriptions" REST resources — scheduled/recurring agent runs, and a pub/sub-ish event subscription mechanism.
  • coworker/cloud — some cloud-facing component (GitHub app installs, Slack workspace linking) even though the core execution is local-first — worth checking how much is actually optional vs. required for OAuth flows that need a public redirect URI.

6. Open questions to explore later

  • How does the permission engine (coworker/permissions) gate destructive actions (file writes, sending Slack messages, etc.) — is there a human-in-the-loop approval UI, and does it map to anything like Claude Code's permission-prompt model?
  • How does aisuite pin/vendoring work in practice, and does it lag upstream aisuite releases in a way that matters?
  • Licensing/governance: MIT, from Andrew Ng's team — worth checking commit cadence and whether it's a serious ongoing OSS project or a launch artifact.
Hermes Wiki