Surface structured error details when sandboxes fail during startup

[NavarrePratt]

Problem

When a sandbox fails to start - due to node resource exhaustion, quota limits, image pull failures, scheduling contention, or other infrastructure issues - the SDK raises generic exceptions with no detail about the cause:

• SandboxTerminatedError("Sandbox {id} was terminated") for sandboxes that reach TERMINATED
• SandboxFailedError("Sandbox {id} failed to start") for sandboxes that reach FAILED

Both are misleading when the sandbox never ran. Users cannot tell whether their code executed at all, and the error message gives no hint about the underlying cause.

This affects any operation that waits for the sandbox to be ready: sandbox.wait(), await sandbox, and auto-waiting operations like exec(), read_file(), and write_file() on a sandbox that hasn't started yet.

Example

with Sandbox.run(container_image="my-image:latest") as sb:
    result = sb.exec(["echo", "hello"]).result()

If the sandbox can't be scheduled (e.g. node has 56 MiB free, sandbox needs 512 MiB), the user sees:

cwsandbox.exceptions.SandboxTerminatedError: Sandbox bfd8ef2e-... was terminated

The actual cause required operator investigation of K8s events to diagnose. With better error surfacing, the message could indicate the scheduling or resource failure.

Backend dependency

The backend is adding structured CWSANDBOX_* error codes to synchronous RPCs (aviato#451), but the async failure path - where the SDK discovers a FAILED/TERMINATED status by polling Get - currently has no way to carry a failure reason. GetSandboxResponse has no reason field. This is tracked in aviato#633.

SDK work in this issue can proceed in parallel on what's possible today, then pick up the async path when the backend is ready.

What needs to happen in the SDK

1. Distinguish "failed to start" from "terminated while running": A sandbox that never reached RUNNING should not raise the same exception as one that was externally stopped mid-execution. The SDK's internal state machine (_Starting -> _Running -> _Terminal) can likely be used to infer this, though there is no explicit "ever reached RUNNING" flag today.

2. Extract structured error details from gRPC status: When the backend returns a grpc.RpcError with ErrorInfo/RetryInfo details containing a CWSANDBOX_* reason, the SDK should parse them and make them available - either in the exception message, as exception attributes (e.g. reason_code, reason_message, retry_delay), or both.

3. Surface status_reason from Get responses (blocked on aviato#633): When the backend adds a reason field to GetSandboxResponse, include it in the exception message and expose it as an attribute on SandboxFailedError/SandboxTerminatedError.

4. Consider retryability: Some backend errors include RetryInfo with suggested retry delays. The SDK could expose this to help users implement retry logic, or handle retries internally for transient errors.

Related

• aviato#633 - Backend: add status_reason to GetSandboxResponse

[nicholaspun-wandb]

Confirmed this still affects the current W&B Sandbox quickstart path.

Reproduction:

from wandb.sandbox import Sandbox

with Sandbox.run(container_image="python:3.15") as sandbox:
    sandbox.exec(["python", "--version"]).result()

Because python:3.15 is not a published image tag, the sandbox reaches FAILED with an underlying ImagePullBackOff / ErrImagePull. The user only receives:

cwsandbox.exceptions.SandboxFailedError:
Sandbox <id> failed to start

The v1 SandboxStatus now contains state_reason, and _SandboxView.status_reason exposes it, so the backend dependency described in this issue appears to be resolved. However, _do_poll_running() still raises SandboxFailedError without including response.status_reason.

Expected behavior:

Sandbox <id> failed to start: ImagePullBackOff: failed to pull image "python:3.15": ...

Ideally, the exception should also expose the reason as structured metadata rather than only embedding it in the message.

This affects exec() and other operations that implicitly wait for the sandbox to become ready. The current traceback is long but contains no actionable startup failure detail.

Separately, the public W&B quickstart currently uses the nonexistent python:3.15 tag and should be corrected to python:3.14 or python:3.15-rc.