cua mcp

Makefile

@@ -6,7 +6,7 @@ MCP_SERVER ?= duckduckgo
 DOCKER_REGISTRY ?= ghcr.io
 DOCKER_REPOSITORY ?= mudler/mcps
 DOCKER_TAG ?= latest
-GO_VERSION ?= 1.25.1
+GO_VERSION ?= 1.26
 
 # Docker image name
 IMAGE_NAME = $(DOCKER_REGISTRY)/$(DOCKER_REPOSITORY)/$(MCP_SERVER)
@@ -20,6 +20,11 @@ ifeq ($(MCP_SERVER),opencode)
 DOCKER_FILE = ./opencode/Dockerfile
 endif
 
+# Override for cua (builds FROM trycua/cua-xfce, not the shared base)
+ifeq ($(MCP_SERVER),cua)
+DOCKER_FILE = ./cua/Dockerfile
+endif
+
 # Default target
 .PHONY: help
 help: ## Show this help message

README.md

@@ -1779,6 +1779,89 @@ mcp:
     }
 ```
 
+### 🖥️ CUA Server
+
+A computer-use server that gives a model a real XFCE desktop and a real Chrome browser inside a container, viewable live over noVNC. It implements no tools of its own: it starts the `computer` and `browser` MCP servers from [nib](https://github.com/mudler/nib) on in-memory transports and re-exposes their merged tool list on a single stdio server.
+
+Unlike every other server in this repository, this one is **container-only**. The binary alone is not useful: it needs the XFCE desktop, the session D-Bus, and the `cua-driver serve` daemon that the image starts inside the desktop session.
+
+**Features:**
+- Desktop control addressed by accessibility element index, not just pixel coordinates (AT-SPI via `cua-driver`)
+- Browser control over an accessibility snapshot with `@eN` element refs
+- Chrome runs on the same desktop, so the browser and desktop tools compose
+- Live view of what the model is doing at `http://localhost:6901`
+- Screenshots are returned as image content, so a vision-capable model is required
+
+**Tools:**
+- `computer_use` - Desktop control. Actions: `capture`, `click`, `double_click`, `right_click`, `middle_click`, `drag`, `scroll`, `type`, `key`, `set_value`, `wait`, `list_apps`, `open_app`, `close_app`, `focus_app`. Capture modes: `som` (numbered elements, default), `vision`, `ax`
+- `browser_navigate` - Open a URL and return a snapshot of the page's interactive elements
+- `browser_snapshot` - Re-read the current page's accessibility tree for fresh refs
+- `browser_click` - Click the element identified by an `@eN` ref
+- `browser_type` - Focus an element, clear it, and type into it
+- `browser_press` - Send a single named key (`Enter`, `Tab`, `Escape`, …) to whatever has focus
+- `browser_scroll` - Scroll the viewport up or down
+- `browser_vision` - Screenshot the current page for visual inspection
+
+**Configuration:**
+- `CUA_ENABLE_COMPUTER` - Register `computer_use` (default: `true`)
+- `CUA_ENABLE_BROWSER` - Register the `browser_*` tools (default: `true`). Setting both this and `CUA_ENABLE_COMPUTER` to `false` is a fatal error
+- `CUA_TOOLS` - Comma-separated list of tools to register, or `all` (default: all)
+- `CUA_DRIVER_CMD` - Path to the `cua-driver` binary (default: `cua-driver`)
+- `CUA_CHROME_PATH` - Chrome binary override (default: nib auto-discovers `/usr/bin/google-chrome`, then `/usr/bin/chromium`)
+- `CUA_BROWSER_PROFILE_DIR` - Chrome profile directory (default: `dante-browser-profile` under the user cache dir, i.e. `/home/cua/.cache/` in this image — it lives and dies with the container unless you mount a volume)
+- `CUA_ALLOW_PRIVATE_URLS` - Allow the browser to navigate to localhost and RFC1918 addresses (default: `false`)
+- `CUA_READY_TIMEOUT` - Budget for the startup readiness gate: X display wait, driver probe, and each upstream handshake (default: `60s`). Values that do not parse, or that are zero or negative, fall back to the default
+- `COGITO_LOG_LEVEL` / `LOG_FORMAT` - nib's log level and format (`json` for JSON). Logs always go to stderr; stdout carries only JSON-RPC
+
+Inherited from the base image, and useful:
+- `VNC_RESOLUTION` - Desktop resolution (default: `1024x768`). Raising it raises the token cost of every screenshot proportionally
+- `VNC_COL_DEPTH` - Colour depth (default: `24`)
+- `VNC_PW` - VNC password. **If unset, the VNC server runs with no authentication at all**
+- `VNC_PORT` / `NOVNC_PORT` - Ports for TigerVNC and noVNC (defaults: `5901`, `6901`)
+
+**Docker Image:**
+```bash
+docker run -i --rm -p 6901:6901 ghcr.io/mudler/mcps/cua:latest
+```
+
+Then open `http://localhost:6901` in a browser to watch the desktop live. Port `5901` is also exposed for a native VNC client; publish it only if you need it, and set `VNC_PW` when you do.
+
+The image is built with `make build MCP_SERVER=cua`, which uses `cua/Dockerfile` rather than the shared one.
+
+**LocalAI configuration (to add to the model config):**
+```yaml
+mcp:
+  stdio: |
+    {
+      "mcpServers": {
+        "cua": {
+          "command": "docker",
+          "args": [
+            "run", "-i", "--rm", "-p", "6901:6901",
+            "ghcr.io/mudler/mcps/cua:master"
+          ]
+        }
+      }
+    }
+```
+
+**Security posture:**
+
+This container is the security boundary, and it is a soft one. Treat it as disposable and keep it isolated.
+
+- **Chrome runs as root with `--no-sandbox`.** The whole container runs as root because supervisord needs it to drop privileges per program, and Chrome refuses to sandbox itself as root, so a wrapper on `/usr/bin/google-chrome` and `/usr/bin/chromium` passes `--no-sandbox`. A renderer compromise therefore yields root inside the container. This is a deliberate, accepted trade-off — dropping Chrome to an unprivileged user costs the shared X session and the AT-SPI tree that `computer_use` depends on.
+- **Do not run this container with `--network host` or host path mounts.** Given the point above, either one turns a browser compromise into a host compromise.
+- **The container is a real interactive desktop.** Anything reachable on its network is reachable by whatever the model drives — which is why `CUA_ALLOW_PRIVATE_URLS` defaults to `false` and why placing this container on a network with internal services deserves thought.
+- **noVNC and VNC are unauthenticated unless `VNC_PW` is set.** Anyone who can reach the published port has full keyboard and mouse control of the desktop. Bind them to localhost or leave them unpublished on shared hosts.
+- nib hard-blocks a small set of key combos (e.g. `cmd+ctrl+q`, `win+l`) and typed-text patterns (`curl … | bash`, `sudo rm -rf`, fork bombs). This is a guardrail against accidents, not a sandbox — there is no interactive approval prompt in this server.
+
+**Notes and limitations:**
+- **The image is large: roughly 6.4 GB**, of which about 5.7 GB is the `trycua/cua-xfce` base. Budget disk and pull time accordingly.
+- **The base image is `trycua/cua-xfce:latest`, unpinned**, as is Google's Chrome apt repository. A rebuild can therefore pick up a different desktop or a different Chrome than the last one did. The Chrome version actually shipped is recorded in the image at `/etc/cua-chrome-version`, and the build fails loudly if the base's `xstartup.sh` changes shape under the daemon injection.
+- **`/dev/uinput` is not available in a container**, so `cua-driver` injects input via `XSendEvent`. Right, middle, and double clicks may not register on some GTK and Qt applications. Clicks addressed by element index go through AT-SPI and are unaffected, which is why element-based addressing is preferred.
+- Startup takes tens of seconds: the desktop, then the accessibility bus, then the `cua-driver serve` daemon must all come up before the first tool call. The entrypoint waits up to 120s for the driver socket.
+- Only `linux/amd64` is built. `trycua` publishes the base for amd64 only, and Google ships no arm64 Chrome `.deb`; an arm64 build fails at the Chrome install step.
+
 ## Development
 
 ### Prerequisites

cua/Dockerfile

@@ -0,0 +1,130 @@
+# The go directive in go.mod is 1.26 (nib requires it), so this cannot be
+# parameterised by the Makefile's GO_VERSION build arg. Pinned directly,
+# matching the root Dockerfile; the unused build arg is ignored by Docker.
+FROM golang:1.26-alpine AS builder
+
+WORKDIR /app
+COPY . .
+RUN CGO_ENABLED=0 go build -o /out/cua ./cua/
+
+# Ubuntu 22.04 (jammy) under the hood, running TigerVNC, noVNC and XFCE from
+# supervisord. glibc 2.35 clears cua-driver's 2.31 floor.
+FROM trycua/cua-xfce:latest
+
+USER root
+
+ARG CUA_DRIVER_VERSION=0.9.1
+# amd64 in practice: trycua publishes cua-xfce for linux/amd64 only, and Google
+# ships no linux/arm64 Chrome deb. The arch mapping below is kept honest rather
+# than load-bearing -- an arm64 build fails at the Chrome apt step, not silently.
+ARG TARGETARCH=amd64
+
+# at-spi2-core and dbus-x11 are what make element-index addressing work: without
+# them cua-driver reports an empty accessibility tree and computer_use falls
+# back to pixel coordinates. Both already ship in the base image, so these are
+# assertions rather than installs -- they keep the dependency explicit if the
+# base ever drops them.
+#
+# Google Chrome comes from Google's own signed apt repository because jammy has
+# no usable chromium: the `chromium` package does not exist there and
+# `chromium-browser` is a transitional stub that redirects to snap, which cannot
+# run in a container. nib probes /usr/bin/google-chrome before /usr/bin/chromium
+# so this satisfies its discovery either way.
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
+        at-spi2-core \
+        dbus-x11 \
+        ca-certificates \
+        curl \
+        gnupg \
+    && curl -fsSL https://dl.google.com/linux/linux_signing_key.pub \
+        | gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg \
+    && echo "deb [arch=${TARGETARCH} signed-by=/usr/share/keyrings/google-chrome.gpg] https://dl.google.com/linux/chrome/deb/ stable main" \
+        > /etc/apt/sources.list.d/google-chrome.list \
+    && apt-get update \
+    && apt-get install -y --no-install-recommends google-chrome-stable \
+    && rm -rf /var/lib/apt/lists/*
+
+# Chrome will not start sandboxed as root in a container and nib does not pass
+# --no-sandbox, so the wrapper takes over both paths nib's discoverChrome
+# probes. /usr/bin/google-chrome arrives from the deb as an update-alternatives
+# symlink; pointing it at the wrapper instead is deliberate.
+#
+# google-chrome-stable is covered too. nib does not probe that name, so nothing
+# depends on it today -- but it is the name the deb advertises and the one a
+# human or a future caller reaches for, and leaving it pointed at the unwrapped
+# binary means that caller gets no --no-sandbox and an opaque launch failure.
+COPY cua/chromium-wrapper.sh /usr/local/lib/chromium-wrapper.sh
+RUN chmod +x /usr/local/lib/chromium-wrapper.sh \
+    && ln -sf /usr/local/lib/chromium-wrapper.sh /usr/bin/google-chrome \
+    && ln -sf /usr/local/lib/chromium-wrapper.sh /usr/bin/chromium \
+    && ln -sf /usr/local/lib/chromium-wrapper.sh /usr/bin/google-chrome-stable
+
+# The apt repo above is unpinned, so record what this build actually shipped;
+# otherwise "which Chrome was in that image" stops being answerable. Running it
+# through the wrapper symlink doubles as a build-time check that the wrapper
+# really does launch Chrome.
+RUN google-chrome-stable --version > /etc/cua-chrome-version \
+    && cat /etc/cua-chrome-version
+
+# cua-driver supplies the AT-SPI element tree and input injection. The
+# `-binary` release asset unpacks a bare `cua-driver` executable.
+RUN case "${TARGETARCH}" in \
+        amd64) CUA_ARCH=x86_64 ;; \
+        arm64) CUA_ARCH=arm64 ;; \
+        *) echo "unsupported TARGETARCH: ${TARGETARCH}" >&2; exit 1 ;; \
+    esac \
+    && curl -fsSL -o /tmp/cua-driver.tar.gz \
+        "https://github.com/trycua/cua/releases/download/cua-driver-rs-v${CUA_DRIVER_VERSION}/cua-driver-rs-${CUA_DRIVER_VERSION}-linux-${CUA_ARCH}-binary.tar.gz" \
+    && tar -xzf /tmp/cua-driver.tar.gz -C /usr/local/bin \
+    && rm /tmp/cua-driver.tar.gz \
+    && chmod +x /usr/local/bin/cua-driver
+
+# Opt out of the driver's default-on telemetry and its start-up update check,
+# which would otherwise call GitHub on every container start. The file is the
+# same shape `cua-driver telemetry disable` writes; it is written directly
+# because `cua-driver config set` needs a running daemon.
+RUN install -d -o cua -g cua /home/cua/.cua-driver \
+    && printf '{\n  "telemetry_enabled": false,\n  "update_check_enabled": false\n}\n' \
+        > /home/cua/.cua-driver/config.json \
+    && chown cua:cua /home/cua/.cua-driver/config.json
+
+# The cua-driver daemon has to start inside the desktop session -- see
+# cua-driver-supervise.sh for why -- which means hooking the base image's
+# xstartup.sh.
+#
+# This *injects* a line rather than shipping our own copy of that file. A
+# verbatim copy would silently win over any future change trycua makes to their
+# xstartup (a new program, different bus handling, a different XFCE
+# invocation): no build error, no runtime error, just a desktop quietly drifted
+# from what the base image expects. Injection inverts that failure mode. The
+# base's xstartup ends with exactly one bare `wait`, and if a future base image
+# no longer does -- none, or more than one, since sed substitutes globally and
+# two anchors would race two daemons over one socket -- the count check takes
+# the build down with a message: loudly, at build time, instead of silently at
+# runtime. This matters more than usual
+# because the FROM above is unpinned `:latest`.
+COPY cua/cua-driver-supervise.sh /usr/local/bin/cua-driver-supervise
+RUN chmod +x /usr/local/bin/cua-driver-supervise \
+    && { [ "$(grep -cx 'wait' /usr/local/bin/xstartup.sh)" = 1 ] \
+        || { echo "ERROR: expected exactly one bare 'wait' line in /usr/local/bin/xstartup.sh" >&2; \
+             echo "to anchor to; found $(grep -cx 'wait' /usr/local/bin/xstartup.sh)." >&2; \
+             echo "The base image's xstartup.sh has changed shape; re-check the injection below." >&2; \
+             echo "Note sed substitutes globally: two anchors would start two daemons." >&2; \
+             exit 1; }; } \
+    && sed -i 's|^wait$|/usr/local/bin/cua-driver-supervise \&\nwait|' /usr/local/bin/xstartup.sh \
+    && [ "$(grep -cx '/usr/local/bin/cua-driver-supervise &' /usr/local/bin/xstartup.sh)" = 1 ]
+
+COPY --from=builder /out/cua /usr/local/bin/cua
+COPY cua/entrypoint.sh /usr/local/bin/entrypoint.sh
+RUN chmod +x /usr/local/bin/entrypoint.sh
+
+ENV DISPLAY=:1
+
+# Stays root, unlike most images here: supervisord owns /var/log/supervisor and
+# /var/run/supervisor.sock and drops to the `cua` user per program, which it
+# can only do as root. The base image runs as root for the same reason.
+
+EXPOSE 5901 6901
+
+ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

cua/aggregate.go

@@ -0,0 +1,71 @@
+package main
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/modelcontextprotocol/go-sdk/mcp"
+)
+
+// upstream is a connected client session to one of nib's in-process MCP
+// servers, together with a label used in error messages.
+type upstream struct {
+	name    string
+	session *mcp.ClientSession
+}
+
+// connectUpstream dials an in-memory transport and completes the MCP
+// handshake, returning a session ready for ListTools and CallTool.
+func connectUpstream(ctx context.Context, name string, t mcp.Transport) (*upstream, error) {
+	c := mcp.NewClient(&mcp.Implementation{Name: "cua-aggregator", Version: version}, nil)
+	sess, err := c.Connect(ctx, t, nil)
+	if err != nil {
+		return nil, fmt.Errorf("connect to %s server: %w", name, err)
+	}
+	return &upstream{name: name, session: sess}, nil
+}
+
+// toolAllowed reports whether name passes the allowlist. An empty allowlist
+// means no filtering.
+func toolAllowed(allow map[string]bool, name string) bool {
+	if len(allow) == 0 {
+		return true
+	}
+	return allow[name]
+}
+
+// aggregate registers every tool exported by ups onto srv, forwarding calls to
+// the owning upstream. Results are passed through verbatim so that image
+// content survives intact.
+func aggregate(ctx context.Context, srv *mcp.Server, ups []*upstream, allow map[string]bool) error {
+	owners := map[string]string{}
+
+	for _, u := range ups {
+		res, err := u.session.ListTools(ctx, nil)
+		if err != nil {
+			return fmt.Errorf("list tools from %s server: %w", u.name, err)
+		}
+
+		for _, tool := range res.Tools {
+			if !toolAllowed(allow, tool.Name) {
+				continue
+			}
+			if owner, dup := owners[tool.Name]; dup {
+				return fmt.Errorf("tool %q is exported by both the %s and %s servers", tool.Name, owner, u.name)
+			}
+			owners[tool.Name] = u.name
+
+			session := u.session // capture per iteration
+			srv.AddTool(tool, func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
+				// Arguments is a json.RawMessage, which marshals verbatim.
+				// Forwarding the raw bytes keeps the request direction as
+				// lossless as the response direction.
+				return session.CallTool(ctx, &mcp.CallToolParams{
+					Name:      req.Params.Name,
+					Arguments: req.Params.Arguments,
+				})
+			})
+		}
+	}
+	return nil
+}