Skip to content

fix(tracing): authenticate to remote viewers, and write traces where they survive - #71

Merged
rdasilveiracabral merged 1 commit into
mainfrom
fix/remote-viewer-trace-auth
Jul 31, 2026
Merged

fix(tracing): authenticate to remote viewers, and write traces where they survive#71
rdasilveiracabral merged 1 commit into
mainfrom
fix/remote-viewer-trace-auth

Conversation

@rdasilveiracabral

Copy link
Copy Markdown
Collaborator

Follow-up to #69. Two remaining reasons a Harbor run on a remote machine produced no usable traces.

1. Trace files were written to a directory that gets deleted

Harbor bind-mounts only /logs/agent and /logs/verifier from the host (harbor.models.trial.paths). Everything else under /logs lives in the container's writable layer, and trial containers are removed by default.

TRACES_DIR pointed at /logs/artifacts/traces, so the exporter worked perfectly and its output was destroyed moments later during cleanup. Observed directly in a running trial:

/logs/agent       -> /dev/nvme0n1p2   (host bind mount, survives)
/logs/artifacts   -> overlay          (container layer, deleted)

Traces now write to /logs/agent/traces/, which is host-mounted — and sit beside trajectory.json, where failure analysis actually looks.

2. Exporters could not authenticate to a non-loopback viewer

The viewer refuses writes from non-loopback clients unless NOOA_VIEWER_AUTH_TOKEN is set, and authenticates via Authorization: Bearer. No exporter sent that header, so remote streaming was impossible however the viewer was configured. Each post failed 403 and retried 3× with backoff, on every LLM call:

POST http://<viewer>:5001/v1/journal/blocks attempt 1/3 failed: HTTP Error 403: Forbidden — retrying in 1s
DROP: failed to export 3 span(s): HTTP Error 403: Forbidden

Adds nooa.tracing._viewer_auth, applied at the three call sites that talk to the viewer: the OTLP HTTP exporter, the litellm journal poster, and the reachability probe. With the token unset, no header is added — loopback development is unchanged.

Verification

Against a token-protected viewer, from a separate host:

request result
POST /v1/traces without header 401
POST /v1/traces with Authorization: Bearer 200

Usage:

# viewer host
NOOA_VIEWER_AUTH_TOKEN=<token> nooa start-dev

# machine running the agent
NOOA_VIEWER_AUTH_TOKEN=<token> OTLP_ENDPOINT=http://<host>:5001/v1/traces

🤖 Generated with Claude Code

…they survive

Two problems stopped a Harbor run on a remote machine from producing usable
traces.

1. Trace files were written to a directory that gets deleted.

   Harbor bind-mounts only /logs/agent and /logs/verifier from the host
   (harbor.models.trial.paths); everything else under /logs lives in the
   container's writable layer, and trial containers are removed by default.
   TRACES_DIR pointed at /logs/artifacts/traces, so every JSONL trace file was
   destroyed during cleanup -- the exporter worked perfectly and the output was
   discarded moments later. Traces now go to /logs/agent/traces/, which is
   host-mounted, and land beside trajectory.json where failure analysis looks.

2. Exporters could not authenticate to a non-loopback viewer.

   The viewer rejects writes from non-loopback clients unless
   NOOA_VIEWER_AUTH_TOKEN is set, and authenticates via
   `Authorization: Bearer` -- but no exporter sent that header, so remote
   streaming was impossible however the viewer was configured. Every span and
   journal post failed 403 and retried three times with backoff, on every LLM
   call.

   Add nooa.tracing._viewer_auth and apply it at the three call sites that talk
   to the viewer: the OTLP HTTP exporter, the litellm journal poster, and the
   reachability probe. When NOOA_VIEWER_AUTH_TOKEN is unset no header is added,
   so loopback development is unchanged.

Verified against a token-protected viewer from a separate host: POST /v1/traces
returns 401 without the header and 200 with it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@rdasilveiracabral
rdasilveiracabral merged commit a5d5bda into main Jul 31, 2026
5 checks passed
@rdasilveiracabral
rdasilveiracabral deleted the fix/remote-viewer-trace-auth branch July 31, 2026 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant