envd: a stale process-stream subscriber can wedge stdout fan-out for all subscribers for ~15 minutes after sandbox resume

[carrotRakko]

Summary

After resuming a paused sandbox, a background process's stdout stream can go completely silent for ~15.5 minutes and then suddenly recover on its own. During the silence, new process.Connect subscribers receive nothing (the start event arrives, but no data events), while unary RPCs (SendInput, List) to the same sandbox work fine.

We traced this to the interaction of three things in packages/envd:

1. MultiplexedChannel.run() fans out to subscribers via unbuffered channel sends (internal/services/process/handler/multiplex.go). A subscriber that is registered but not reading blocks the fan-out loop for every subscriber. The done-guard only helps for cancelled subscribers.
2. A subscriber's consumer goroutine is only cleaned up when stream.Send returns an error (internal/services/process/connect.go, start.go). There is no write deadline on the response stream, so a send to a dead peer blocks until the guest kernel's TCP gives up retransmitting — with the default net.ipv4.tcp_retries2 = 15, that is ~924 seconds ≈ 15.4 min, which matches our observed silence duration exactly.
3. Pausing snapshots the whole VM, including the TCP state of streams whose peer-side close has not yet been processed by envd. Even when the client calls disconnect() before pause() (we do), the FIN may not reach envd before the snapshot is taken. The snapshot then preserves a "ghost" subscriber + consumer goroutine that believes its connection is alive.

After resume, the ghost's stream.Send blocks on a connection whose remote side no longer exists; the fan-out loop blocks on the ghost's unbuffered channel; every legitimate subscriber (including ones attached after resume via process.Connect) starves. When the guest TCP stack finally gives up (~924s), the write errors out, the ghost is removed (defer dataCancel()), and buffered events suddenly flow to the healthy subscribers.

Observed impact (production)

We run agent workloads that pause on human-in-the-loop waits and resume on reply (Python SDK, betaPause/connect). Incident pattern, observed on 3 independent sandboxes in one day:

• resume completes; commands.connect(pid, ...) re-attaches and receives the start event
• 0 or 1 data events arrive (consistent with whether the ghost was frozen inside Send or waiting on <-data; the first event after resume can be absorbed by the ghost's socket send buffer)
• then total silence on stdout for ~16 min, while SendInput / List on the same sandbox keep working (separate unary connections)
• stdout suddenly resumes and the workload completes normally

Reproduction

Honest note: we could not reproduce this on demand (4 attempts: with/without client disconnect() before pause, immediate and delayed resume, large payloads to fill socket buffers). In all attempts the peer close apparently reached envd before the snapshot, so no ghost was created. We suspect triggering depends on timing and possibly on whether resume placement changes the network path, which the client cannot control. The production evidence (three incidents with the exact tcp_retries2 timeout signature, and the 0-vs-1-event split matching the two possible frozen states of the consumer goroutine) is what points at this mechanism.

Suggested fixes

Either (or both) of:

1. Set a write deadline on stream sends in the process service consumers, e.g. via http.ResponseController.SetWriteDeadline, keyed to the keepalive interval the client already sends (Keepalive-Ping-Interval header). A dead peer then errors out in seconds instead of ~15 minutes, and the existing cleanup path (cancel + defer dataCancel()) takes over.
2. Make the fan-out robust to a slow/stuck subscriber in MultiplexedChannel.run(): e.g. per-subscriber buffered channel with "cancel the subscriber on overflow" semantics, so one stuck consumer can never starve the others. (Dropping data silently would be wrong for stdout; disconnecting the stuck subscriber is consistent with what already happens when Send errors.)

Workaround we deployed

Setting net.ipv4.tcp_retries2 = 8 (RFC 1122's lower bound, ~100s) inside the guest right after sandbox creation shortens the wedge from ~15 min to ~1-2 min. Since sysctl state is part of the VM memory snapshot, it survives pause/resume. This is a mitigation, not a fix — the fan-out still wedges, just for less time.

Environment

• e2b Python SDK 2.x (AsyncSandbox, betaPause / connect)
• Custom templates based on python:3.14-slim / node:24-slim / debian:13-slim
• E2B cloud (us), Firecracker sandboxes with pause/resume

Happy to provide timestamps/sandbox IDs privately if useful.

---

This analysis was drafted with an AI coding agent; the code reading, production log correlation, and the deployed workaround were verified by our team.