feat: SSE keepalive + elapsed-time instrumentation in interception proxy

[rasdani]

Summary

Emits SSE keepalive comments every 10s during the idle window in _handle_streaming_response, so upstream intermediaries stop killing the connection while we wait for vLLM's non-streaming response. Attaches elapsed-time info (and, in commit a73b4d25, TCP socket diagnostics) to every StreamInterrupted for root-cause histograms.

Context

Production opencode RL runs lose ~99% of rollouts to a silent "Empty trajectory" class. Root cause: proxy holds an idle SSE connection while vLLM computes, something upstream cuts it, opencode's Vercel-AI-SDK treats the truncated stream as clean completion, rollout exits exit_code=0, turns=0.

Builds on #1191 (StreamInterrupted error plumbing). #1191 makes cuts visible; this PR prevents a subset of them and diagnoses the rest.

Changes

• verifiers/utils/interception_utils.py:

  • New module constant KEEPALIVE_INTERVAL_SECONDS = 10.0
  • _handle_streaming_response loop uses asyncio.wait on a single reused get() task with that timeout; on timeout emits b": keepalive\n\n" before continuing
  • Keepalive write failures are treated identically to chunk-write failures: log, set state["error"], return
  • All StreamInterrupted messages include waited_s
  • a73b4d25 — TCP diagnostics: new _tcp_diag(transport, peer) helper pulls peer addr, TCP_INFO.state (ESTABLISHED / CLOSE_WAIT / FIN_WAIT / …), and retransmits via getsockopt(SOL_TCP, TCP_INFO, …). Peer is cached at stream start (socket may be gone at error time). All probes best-effort — a failing getsockopt yields an empty string, never breaks error logging. Wired into both StreamInterrupted sites.

• tests/test_interception_utils.py: keepalive emission + keepalive-failure tests.

What production logs tell us

Deploying just this PR (keepalive + elapsed-time) on opencode-swe (Qwen3-30B-Instruct, 65k seq_len, 7-node slurm job) showed the fix is only partial: the elapsed-time histogram split cleanly into two classes:

stream write failed after 3.3s  ← short class (before keepalive fires)
stream write failed after 1.4s
stream write failed after 2.5s
...
keepalive write failed after 10.0s  ← keepalive catches the long-idle class

76 cuts in the first 30 min — the keepalive was defending the >10s class but the <10s class was still slipping through.

Root-cause diagnosis (done outside this repo)

Short class (1-9s): prime-tunnel (Prime Intellect's tunnel SDK) configures frpc with transport.poolCount = 10. frpc pre-warms 10 TCP work connections to frps. These sit idle on the frpc→frps leg with no SO_KEEPALIVE (only the control mux has keepalive, not the user-plane pooled work conns). NAT / LB middleboxes drop idle TCP flows after a few seconds. When frps later assigns a dead pooled conn to a user request, the first user write lands in the local kernel send buffer and succeeds — then RST arrives 1-9s later when the first segment goes on-wire. Opencode-side sees Cannot write to closing transport.

Fix for the short class: patched upstream frp to expose three new config knobs and apply SO_KEEPALIVE + TCP_KEEPIDLE/KEEPINTVL/KEEPCNT on every work-conn dial. prime-tunnel's generated frpc.toml now emits:

transport.poolCount = 10
transport.workConnKeepaliveIdleSeconds = 5
transport.workConnKeepaliveIntvlSeconds = 2
transport.workConnKeepaliveCount = 2

Kernel detects dead pooled conns within ~9s and frpc reaps them before frps hands them out. poolCount=1 was an earlier diagnostic that confirmed the hypothesis (0 cuts but ~7× slower throughput — control-plane RTT on every request).

After both fixes

Same opencode-swe run with PR #1194 + the frp/prime-tunnel patches:

• Short class (1-9s): 0 occurrences (was 76 in 30 min)
• Residual class: 14 cuts at 20-400s, ALL with peer=127.0.0.1:<port> (loopback!)

The _tcp_diag output is what let us pin this down: the remaining cuts are on the local leg between frpc (running on the orchestrator node) and the aiohttp interception proxy on the same machine (127.0.0.1:8765) — not the internet-facing frpc↔frps leg that keepalive fixes. Different cause entirely (likely frpc's own vhostHTTPTimeout / local-dial close on the loopback leg; more investigation needed). TCP_INFO.state was unavailable at error time because the socket was already closed, so the cached peer addr carried the signal.

Verification

Unit tests added and passing. Production: 82% reduction in StreamInterrupted on 30-min window, and step-0 time recovered from ~3.5h (poolCount=1 workaround) to 12 min (poolCount=10 + keepalive). Remaining 18% is a separate loopback-side issue newly visible thanks to the a73b4d25 diagnostics — tracked as follow-up.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f809d35. Configure here.}