MCP gateway (gh-aw-mcpg) emits spans with an empty OTel Resource (no service.name) — resource detection fails on semconv schema-URL conflict, breaking trace stitching

[yskopets]

🤖 This issue has been generated by Claude Code.

Summary

The MCP gateway (ghcr.io/github/gh-aw-mcpg) emits its spans (gateway.request, mcp.tool_call, gateway.backend.execute) with an empty OpenTelemetry Resource — no service.name, no attributes at all — because resource detection fails at gateway startup with a semconv schema-URL conflict. As a result, even though the gateway spans share the correct traceId and correctly point to the agent.setup span as their parent, observability backends that assemble/segment the waterfall per service cannot stitch them into the gh-aw trace. The gh-aw lifecycle spans render as one tree, and the gateway spans render as a separate, parent-less group of orphan roots.

Tested with gh-aw CLI v0.79.1, engine Claude Code 2.1.168, awf v0.25.68, gateway image ghcr.io/github/gh-aw-mcpg:v0.3.25.

(Note: the OTLP spans report this under an attribute literally named gh-aw.cli.version: 2.1.168, but per the run's aw_info.json that 2.1.168 is the Claude Code engine version — cli_version is v0.79.1. The gh-aw.cli.version span attribute therefore appears to carry the engine/agent version rather than the gh-aw CLI version. Tracked separately in #38897.)

Root cause: gateway resource detection fails → empty Resource

At gateway startup the OTLP tracer logs:

[tracing:provider] Initializing OTLP tracing: endpoint=https://<otlp-endpoint>/v1/traces, service=mcp-gateway, sampleRate=1.00
[tracing:provider] Warning: failed to create OTEL resource: error detecting resource: conflicting Schema URL: https://opentelemetry.io/schemas/1.34.0 and https://opentelemetry.io/schemas/1.41.0
[tracing:provider] OTLP tracing initialized successfully

The gateway intends service.name=mcp-gateway, but the schema-URL conflict (OTel semantic-conventions 1.34.0 vs 1.41.0 inside the gh-aw-mcpg binary) discards the entire Resource, so spans are exported with no service identity.

Evidence (OTLP/JSON received by a backend)

Same run, single OTLP endpoint, all spans received. Comparing the resourceSpans[].resource of the two span groups:

// gh-aw CLI lifecycle spans  (scope "gh-aw")
"resource": { "attributes": [
  {"key":"service.name","value":{"stringValue":"gh-aw.<workflow>"}},
  // …23 attributes total: gh-aw.run.id, github.repository, runner.*, deployment.environment, …
]}

// MCP gateway spans  (scope "github.com/github/gh-aw-mcpg")
"resource": {}      // ← completely empty: no service.name, no attributes

Crucially, the linkage itself is correct — this is purely a Resource problem, not a trace/parent problem:

Check	gh-aw spans	gateway (mcpg) spans
traceId	… (X)	same (X) ✅
gateway.request.parentSpanId	—	= agent.setup's spanId ✅
project routing header (x-project-name)	present	same value ✅
resource.service.name	gh-aw.<workflow>	absent (empty resource) ❌

So all 28 gateway.request spans + the mcp.tool_call → gateway.backend.execute branch are parented to agent.setup, in the same trace and same project — they only differ by Resource.

Impact

Because the gateway spans carry no service.name, backends that group the trace by service (and span/LLM-observability UIs in particular) place them in a separate unknown_service bucket. Their shared parent (agent.setup) lives in the gh-aw.<workflow> service bucket and is therefore "missing" from the gateway group, so the gateway spans appear as a flat set of orphan roots rather than as children of agent.setup.

Net effect for the user: instead of one connected trace, you see:

• Tree 1 — the gh-aw lifecycle spans (pre_activation → activation → agent → detection → safe_outputs → conclusion).
• Tree 2 — a disconnected pile of gateway.request / mcp.tool_call spans with no visible parent.

The actual tool-call telemetry (which model/tool the agent invoked) is thus detached from the run it belongs to.

Expected behavior

The gateway should emit a valid OTel Resource with at least service.name, so its spans stitch under agent.setup within the same trace. Ideally it would also carry the correlation attributes the gh-aw CLI spans already set (e.g. gh-aw.run.id, gh-aw.repository, github.run_id) so the two services correlate cleanly even in UIs that lane by service.

Suggested fix

1. Resolve the OTel semconv schema-URL conflict (1.34.0 vs 1.41.0) in gh-aw-mcpg so resource detection no longer fails (align the semconv dependency versions, or construct the Resource without the conflicting auto-detector / with an explicit schemaURL).
2. Ensure the gateway always sets service.name (e.g. mcp-gateway) even if auto-detection partially fails — a failed detector should not wipe an explicitly-provided service.name.
3. Optionally propagate the gh-aw run/repo correlation attributes into the gateway Resource.

Questions

• Is the empty-Resource fallback on detector failure intended, or should an explicit service.name survive?
• Is the 1.34.0/1.41.0 schema-URL conflict known/tracked?

I can share the redacted OTLP capture and full gateway log on request.

---

Note: observed in a private repository. Endpoint URLs, repo/workflow names, span IDs, and auth headers have been redacted; the gateway log line is quoted verbatim except for the endpoint host.

[lpcox]

Tracking issue created in gh-aw-mcpg: https://github.com/github/gh-aw-mcpg/issues/github/gh-aw-mcpg#7492

This covers the root cause (semconv schema-URL conflict in internal/tracing/provider.go causing fallback to resource.Empty()) and proposes both an immediate safe fix (non-empty fallback preserving service.name) and a root-cause fix (aligning the semconv import version with OTel SDK v1.44.0).

Generated by Gateway Issue Dispatcher · 470.2 AIC · ⊞ 32.8K · ◷