feat: Add OpenTelemetry tracing support to api-proxy sidecar

[lpcox]

Summary

Add OpenTelemetry (OTEL) tracing to the api-proxy sidecar container so that every proxied LLM API request emits a span with token usage data, provider info, and request metadata. Spans should be children of the parent workflow trace (via GITHUB_AW_OTEL_TRACE_ID / GITHUB_AW_OTEL_PARENT_SPAN_ID env vars), enabling end-to-end distributed tracing from the GitHub Actions workflow through MCPG and into the api-proxy.

Motivation

The api-proxy already captures rich telemetry:

• Token usage (input/output/cache tokens, model, provider, duration) → token-usage.jsonl
• In-memory metrics (counters, histograms) → /metrics endpoint
• Structured logs (JSON to stdout)

However, none of this data participates in the distributed trace that gh-aw establishes for each workflow run. The MCPG already exports spans via OTLP/HTTP. Adding OTEL to the api-proxy would complete the observability picture.

How OTEL Works in the gh-aw Ecosystem Today

1. Workflow frontmatter declares observability.otlp with endpoint + headers
2. gh-aw compiler generates env vars: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME
3. gh-aw setup action generates trace context: GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID
4. MCPG receives config via stdin JSON (gateway.opentelemetry.{endpoint, traceId, spanId}) and exports spans via Go OTLP/HTTP exporter
5. Logs/errors written to /tmp/gh-aw/otel.jsonl and /tmp/gh-aw/otlp-export-errors.jsonl

Reference: smoke-otel-tracing.md and its compiled lock file

Proposed Architecture

┌──────────────────────────────────────────────────────────┐
│ api-proxy (Node.js)                                      │
│                                                          │
│  Env vars:                                               │
│    OTEL_EXPORTER_OTLP_ENDPOINT                           │
│    OTEL_EXPORTER_OTLP_HEADERS                            │
│    OTEL_SERVICE_NAME = "awf-api-proxy"                   │
│    GITHUB_AW_OTEL_TRACE_ID                               │
│    GITHUB_AW_OTEL_PARENT_SPAN_ID                         │
│                                                          │
│  New module: otel.js                                     │
│    - SDK initialization + tracer singleton               │
│    - W3C TraceContext parent propagation from env vars   │
│    - OTLP/HTTP exporter (through Squid proxy)            │
│    - File-based fallback when no endpoint configured     │
│                                                          │
│  Integration points:                                     │
│    proxy-request.js → create span per request            │
│    token-tracker-http.js → add token attrs to span       │
│    server.js → graceful shutdown (flush spans)           │
└──────────────────────────────────────────────────────────┘
         │
         ▼ OTLP/HTTP (routed through Squid proxy)
   [Sentry / Datadog / OTEL Collector]

Span Design

Each proxied request becomes a span:

Field	Value
Span name	api_proxy.{provider}.request
Kind	CLIENT
Parent	Constructed from GITHUB_AW_OTEL_TRACE_ID + GITHUB_AW_OTEL_PARENT_SPAN_ID

Span Attributes

Following GenAI Semantic Conventions:

Attribute	Source
gen_ai.system	Provider name (openai, anthropic, copilot)
gen_ai.request.model	Model from request body
gen_ai.response.model	Model from response
gen_ai.usage.input_tokens	From token tracker
gen_ai.usage.output_tokens	From token tracker
awf.cache_read_tokens	From token tracker
awf.cache_write_tokens	From token tracker
http.request.method	GET/POST
http.response.status_code	Upstream status
url.path	Request path
awf.request_id	Internal request ID
awf.streaming	boolean
awf.provider	Provider name

Span Events

• gen_ai.usage — emitted at span end with full token breakdown
• rate_limit — when rate limiting is triggered
• error — on upstream errors

Implementation Plan

Phase 1: Core OTEL module + span-per-request

1. Add dependencies to containers/api-proxy/package.json:

   • @opentelemetry/api (~50KB)
   • @opentelemetry/sdk-trace-node (~100KB)
   • @opentelemetry/exporter-trace-otlp-http (~50KB)
   • @opentelemetry/resources + @opentelemetry/semantic-conventions

2. Create containers/api-proxy/otel.js:

   • Read env vars for endpoint/headers/service name
   • Construct parent context from GITHUB_AW_OTEL_TRACE_ID + GITHUB_AW_OTEL_PARENT_SPAN_ID
   • Initialize NodeTracerProvider with BatchSpanProcessor + OTLP exporter
   • Export getTracer(), shutdown(), isEnabled() functions
   • OTLP exporter must use HTTPS_PROXY agent (route through Squid)

3. Integrate into proxy-request.js:

   • Start span at request begin (where requestId is generated)
   • Set attributes from request metadata
   • End span in the response end handler
   • Set span status on error

4. Integrate into token-tracker-http.js:

   • In finalizeTracking(), add token usage attributes to active span
   • Use the existing onUsage callback hook

5. Forward env vars from src/services/api-proxy-service.ts:

   • Pass OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME
   • Pass GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID

6. Graceful shutdown in server.js:

   • Call otel.shutdown() in the existing shutdown handler to flush pending spans

Phase 2: File fallback + tests

7. File-based fallback: When no OTLP endpoint is configured, write span data as NDJSON to /var/log/api-proxy/otel.jsonl (mirrors MCPG pattern)

8. Tests: Unit tests for otel.js (mock exporter), integration with proxy-request.js

Phase 3: Extended observability

9. Additional spans: Model resolution, OIDC token refresh, rate limit decisions
10. Metrics via OTEL (optional): Replace or augment in-memory metrics with OTEL metrics exporter

Design Decisions

Decision	Choice	Rationale
SDK	@opentelemetry/sdk-trace-node	Standard Node.js OTEL SDK
Exporter	OTLP/HTTP	Works through Squid proxy (HTTPS)
Parent context	Env vars	Matches MCPG pattern; spans are children of workflow trace
Span granularity	One per proxy request	Natural boundary; maps to existing proxyRequest() lifecycle
Token data	Span attributes	GenAI semantic conventions; queryable in trace backends
Fallback	NDJSON file	Matches /tmp/gh-aw/otel.jsonl pattern from MCPG
Activation	Opt-in via OTEL_EXPORTER_OTLP_ENDPOINT presence	Zero overhead when not configured
Proxy routing	OTLP exporter uses HTTPS_PROXY env	Already set in api-proxy container

Open Questions

1. Should the api-proxy spans be direct children of the workflow parent span, or should they be children of the MCPG tool-call spans? (The MCPG is the one invoking the LLM via the proxy)
2. Should we also export OTEL metrics (not just traces)? The api-proxy already has a Prometheus-style /metrics endpoint.
3. What is the minimum Node.js version for the OTEL SDK? (Currently requires >=14, api-proxy requires >=18, so no issue)
4. Should the token-usage.jsonl file be deprecated in favor of OTEL spans long-term, or kept as a complementary local record?

Estimated Effort

• Phase 1 (core): ~2-3 days
• Phase 2 (fallback + tests): ~1-2 days
• Phase 3 (extended): ~1-2 days

Files to Modify

• containers/api-proxy/package.json — add OTEL dependencies
• containers/api-proxy/otel.js — new: SDK init, tracer, shutdown
• containers/api-proxy/proxy-request.js — span creation/completion
• containers/api-proxy/token-tracker-http.js — add token attrs to span
• containers/api-proxy/server.js — graceful shutdown integration
• src/services/api-proxy-service.ts — forward OTEL env vars to container