fix(computer-use): serialize X11 screenshot access

[Tushar-Khandelwal-2004]

Summary

Fixes #5047.

Serialize X11 screenshot/display access through a shared mutex and replace the Linux GetWindows robotgo title lookup with direct X11 window enumeration.

What Changed

• Added a shared X11 client gate for screenshot, compressed screenshot, display info, and window enumeration paths.
• Replaced the Linux GetWindows implementation with direct xgbutil/ewmh enumeration. It now opens a single X11 connection per call, defers xu.Conn().Close() immediately, reads the client list, resolves titles and PIDs via ewmh, deduplicates by PID, and returns the same WindowInfo shape as the original robotgo path.
• Moved full-screen and region screenshot capture through common helper functions so X11 access is handled consistently.
• Refactored normal and compressed screenshot encoding to share the cursor-safe capture/encoding flow.
• Non-Linux keeps the existing robotgo path untouched.

Why

Repeated calls to /computeruse/display/windows could monotonically increase X11 client count because robotgo.GetTitle opens xgbutil connections internally on Linux without closing them. Serialization alone did not fix this since the leak is monotonic even without concurrency. The complete fix owns the X11 window listing directly so connection lifecycle is deterministic.

Testing

• Ran compile-only package check in Docker:

docker run --rm -v ${PWD}:/repo -w /repo golang:1.25-bookworm bash -lc "apt-get update >/dev/null && apt-get install -y libx11-dev libxtst-dev libxinerama-dev gcc pkg-config >/dev/null && /usr/local/go/bin/go build ./libs/computer-use/..."

[cubic-dev-ai]

2 issues found across 5 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="libs/computer-use/pkg/computeruse/display.go">

<violation number="1" location="libs/computer-use/pkg/computeruse/display.go:106">
P2: X11 serialization lock held across expensive non-X11 CPU work (image encoding, base64)</violation>
</file>

<file name="libs/computer-use/pkg/computeruse/screenshot_capture.go">

<violation number="1" location="libs/computer-use/pkg/computeruse/screenshot_capture.go:25">
P2: Primary-display bounds cache can become stale across display restarts/geometry changes, so full-screen screenshots may capture with outdated coordinates.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Primary-display bounds cache can become stale across display restarts/geometry changes, so full-screen screenshots may capture with outdated coordinates.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At libs/computer-use/pkg/computeruse/screenshot_capture.go, line 25:

<comment>Primary-display bounds cache can become stale across display restarts/geometry changes, so full-screen screenshots may capture with outdated coordinates.</comment>

<file context>
@@ -0,0 +1,65 @@
+}
+
+func capturePrimaryDisplay() (*image.RGBA, error) {
+	if !primaryDisplayBoundsCached || primaryDisplayBounds.Empty() {
+		primaryDisplayBounds = screenshot.GetDisplayBounds(0)
+		primaryDisplayBoundsCached = true
</file context>

[nx-cloud]

View your CI Pipeline Execution ↗ for commit 98779fe

Command	Status	Duration	Result
nx e2e:cleanup daytona-e2e	✅ Succeeded	<1s	View ↗
nx run-many --target=test:e2e --all --nxBail=true	✅ Succeeded	6m 34s	View ↗
nx e2e daytona-e2e	✅ Succeeded	31s	View ↗
nx run-many --target=build --projects=api,runne...	✅ Succeeded	13s	View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗

---

☁️ Nx Cloud last updated this comment at 2026-06-16 18:04:35 UTC

[mu-hashmi]

@cubic-dev-ai review

[cubic-dev-ai]

@cubic-dev-ai review

@mu-hashmi I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 4 files

_{Re-trigger cubic}

[rovle]

@Tushar-Khandelwal-2004 could you please update the PR body to match the current diff? It still mentions caching primary display bounds and resetting that cache on Start/Stop, but the current commit doesn’t include that cache/reset logic.

[Tushar-Khandelwal-2004]

Thanks for catching that @rovle. I updated the PR body so it matches the current diff and removed the stale wording.

[rovle]

I looked a bit into this and I don’t think this PR fully fixes the reported X11 client exhaustion. I reproduced the issue through the real Computer Use HTTP wrappers + real ComputerUse implementation by repeatedly calling /computeruse/display/windows before another /computeruse/screenshot?showCursor=true, and on both main and this PR current head (98779fec6), Xvfb client/socket count still grows to the max-client cap and the follow-up screenshot fails:

baseline xvfb socket_fds=5
after windows call 01 xvfb socket_fds=114 delta=+109
after windows call 02 xvfb socket_fds=130 delta=+125
...
screenshot_status=400
body={"error":"x protocol authentication refused: Maximum number of clients reached"}

The remaining leak appears to be in GetWindows: this PR serializes that path with withX11Client, but it still calls robotgo.FindIds("") followed by robotgo.GetTitle(id). On Linux, robotgo.GetTitle(id) goes through robotgo_x11.go; when resolving a PID to an X window, GetXid can call xgbutil.NewConn(), and that connection is not closed. Serialization prevents concurrent entry, but it does not fix monotonic client growth.

So unless I missed something, my read is that this PR is a concurrency mitigation for screenshot/display access, but not a complete fix for the reported X11 client exhaustion. A complete fix should avoid that robotgo.GetTitle path or ensure the X11 connection it opens is explicitly closed, and should include a regression test that repeated /computeruse/display/windows calls do not increase Xvfb client count or break a follow-up screenshot.

[Tushar-Khandelwal-2004]

Good catch. The serialization only prevented concurrent leaks but didn't fix the monotonic growth per call. The root cause is robotgo.GetTitle opening an xgbutil.NewConn() internally per window title lookup with no close. Working on replacing that path with a single explicit xgbutil connection that enumerates windows and titles in one shot and defers close, targeting the /computeruse/display/windows repro directly. Will push shortly.

[rovle]

Will take a closer look later, but just a note that currently this PR changes the semantics of GetWindows. The ID it returned used to be the PID, this PR would change it to be the X11 window ID. It also changes some subtler semantics on which windows appear (old was at most one titled window per process, now possibly multiple etc.).

[Tushar-Khandelwal-2004]

yeah my bad, restored both ID is PID again via ewmh.WmPidGet and deduplicated by PID with a seen map so only one titled window per process shows up, same as before.

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="libs/computer-use/pkg/computeruse/getwindows_linux.go">

<violation number="1" location="libs/computer-use/pkg/computeruse/getwindows_linux.go:48">
P0: `WindowInfo.ID` is set to the process PID instead of the X11 window ID (`win`), causing multi-window apps to be silently deduplicated and the returned ID to be semantically wrong (PID, not a window handle).</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P0: WindowInfo.ID is set to the process PID instead of the X11 window ID (win), causing multi-window apps to be silently deduplicated and the returned ID to be semantically wrong (PID, not a window handle).

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At libs/computer-use/pkg/computeruse/getwindows_linux.go, line 48:

<comment>`WindowInfo.ID` is set to the process PID instead of the X11 window ID (`win`), causing multi-window apps to be silently deduplicated and the returned ID to be semantically wrong (PID, not a window handle).</comment>

<file context>
@@ -37,8 +43,9 @@ func getWindowsX11() ([]computeruse.WindowInfo, error) {
+		seen[pid] = true
 		windows = append(windows, computeruse.WindowInfo{
-			ID:    int(win),
+			ID:    int(pid),
 			Title: title,
 			Position: computeruse.Position{
</file context>