fix(tracing): authenticate to remote viewers, and write traces where they survive - #71
Merged
Merged
Conversation
…they survive Two problems stopped a Harbor run on a remote machine from producing usable traces. 1. Trace files were written to a directory that gets deleted. Harbor bind-mounts only /logs/agent and /logs/verifier from the host (harbor.models.trial.paths); everything else under /logs lives in the container's writable layer, and trial containers are removed by default. TRACES_DIR pointed at /logs/artifacts/traces, so every JSONL trace file was destroyed during cleanup -- the exporter worked perfectly and the output was discarded moments later. Traces now go to /logs/agent/traces/, which is host-mounted, and land beside trajectory.json where failure analysis looks. 2. Exporters could not authenticate to a non-loopback viewer. The viewer rejects writes from non-loopback clients unless NOOA_VIEWER_AUTH_TOKEN is set, and authenticates via `Authorization: Bearer` -- but no exporter sent that header, so remote streaming was impossible however the viewer was configured. Every span and journal post failed 403 and retried three times with backoff, on every LLM call. Add nooa.tracing._viewer_auth and apply it at the three call sites that talk to the viewer: the OTLP HTTP exporter, the litellm journal poster, and the reachability probe. When NOOA_VIEWER_AUTH_TOKEN is unset no header is added, so loopback development is unchanged. Verified against a token-protected viewer from a separate host: POST /v1/traces returns 401 without the header and 200 with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #69. Two remaining reasons a Harbor run on a remote machine produced no usable traces.
1. Trace files were written to a directory that gets deleted
Harbor bind-mounts only
/logs/agentand/logs/verifierfrom the host (harbor.models.trial.paths). Everything else under/logslives in the container's writable layer, and trial containers are removed by default.TRACES_DIRpointed at/logs/artifacts/traces, so the exporter worked perfectly and its output was destroyed moments later during cleanup. Observed directly in a running trial:Traces now write to
/logs/agent/traces/, which is host-mounted — and sit besidetrajectory.json, where failure analysis actually looks.2. Exporters could not authenticate to a non-loopback viewer
The viewer refuses writes from non-loopback clients unless
NOOA_VIEWER_AUTH_TOKENis set, and authenticates viaAuthorization: Bearer. No exporter sent that header, so remote streaming was impossible however the viewer was configured. Each post failed 403 and retried 3× with backoff, on every LLM call:Adds
nooa.tracing._viewer_auth, applied at the three call sites that talk to the viewer: the OTLP HTTP exporter, the litellm journal poster, and the reachability probe. With the token unset, no header is added — loopback development is unchanged.Verification
Against a token-protected viewer, from a separate host:
POST /v1/traceswithout headerPOST /v1/traceswithAuthorization: BearerUsage:
🤖 Generated with Claude Code