Skip to content

Responses API proxy hangs when using ARC runners with k8s #108

Description

@almssp

Summary

The Start Responses API proxy step backgrounds codex-responses-api-proxy with ( ... ) & and does not redirect its stdout/stderr. On self-hosted runners that deliver steps into a sidecar container via a WebSocket exec — notably Actions Runner Controller (ARC) in kubernetes container mode — the proxy then holds the parent step's exec stream open after bash exits. The next step's hook hangs until the websocket heartbeat times out (~1m52s in our environment) and the runner reports Executing the custom container implementation failed.

GitHub-hosted runners are unaffected (each step is a direct subprocess of the runner agent, so a backgrounded child holding stdio doesn't hold any exec stream open).

Symptom (v1.8)

##[group]Run (
  printenv PROXY_API_KEY | env -u PROXY_API_KEY "${args[@]}"
) &
...
responses-api-proxy listening on 127.0.0.1:NNNNN
[ ~1m52s of silence — proxy is up, no further step output ]
##[error]Executing the custom container implementation failed. Please contact your self hosted runner administrator.

In the runner pod's worker log, the hook process for the next step starts but never writes its response file. The 1m52s gap matches ARC's WebSocket heartbeat defaults (pingPeriodMs=5000, pongDeadlineMs=ping*12+1000≈61s) plus close-timeout / setup overhead.

Root cause

action.yml (v1.8), Start Responses API proxy step:

(
  printenv PROXY_API_KEY | env -u PROXY_API_KEY "${args[@]}"
) &

The subshell inherits the parent step's stdout/stderr. When bash exits, the backgrounded proxy keeps those FDs open for the lifetime of the job. On ARC k8s mode those FDs are the WebSocket exec stream, so the hook stays blocked.

Proposed fix

Two-line change. Detaches stdio while preserving proxy output in a log file under $RUNNER_TEMP for diagnostics. No behaviour change on github-hosted.

 (
   printenv PROXY_API_KEY | env -u PROXY_API_KEY "${args[@]}"
-) &
+) </dev/null >>"${RUNNER_TEMP:-/tmp}/codex-responses-api-proxy.log" 2>&1 &
+disown

Happy to send the PR.

Workaround we're using today

Runtime patch in the calling workflow, before invoking the action. The manifest is at _actions/openai/codex-action/<ref>/action.yml on the runner pod's work volume (mounted at /__w/_actions/… inside the $job container in ARC k8s mode):

- name: Patch openai/codex-action proxy step (ARC stdio detach)
  run: |
    set -euo pipefail
    action_yml=/__w/_actions/openai/codex-action/v1.8/action.yml
    [ -f "$action_yml" ] || { echo "::error::$action_yml missing" >&2; exit 1; }
    grep -qF 'codex-responses-api-proxy.log' "$action_yml" && { echo "already patched"; exit 0; }
    awk '
      BEGIN { in_block = 0 }
      /printenv PROXY_API_KEY \| env -u PROXY_API_KEY/ { in_block = 1; print; next }
      in_block && /^        \) &$/ {
        print "        ) </dev/null >>\"${RUNNER_TEMP:-/tmp}/codex-responses-api-proxy.log\" 2>&1 &"
        print "        disown"
        in_block = 0; next
      }
      { print }
    ' "$action_yml" > "$action_yml.new"
    grep -qF 'codex-responses-api-proxy.log' "$action_yml.new" \
      || { rm -f "$action_yml.new"; echo "::error::upstream shape changed; refresh patch" >&2; exit 1; }
    mv "$action_yml.new" "$action_yml"

- name: Run Codex review
  uses: openai/codex-action@v1.8
  with:
    # ...

We'd much rather drop this step.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions