Agent execution step lands in a different OTel trace than the job-level instrumentation (instrument/job) establishes

[plengauer]

Summary

In the agent job of the Autotriage workflow, the trace that plengauer/opentelemetry-github/actions/instrument/job (job-level instrumentation) establishes for the job is not the trace that ends up used by the agentic workflow's own built-in observability (GITHUB_AW_OTEL_TRACE_ID / GITHUB_AW_OTEL_PARENT_SPAN_ID, consumed by the "Execute GitHub Copilot CLI" step). The job ends up with two disconnected root traces instead of one continuous trace.

Job log: https://github.com/plengauer/Thoth/actions/runs/30767171258/job/91547886181 (run 30767171258, job 91547886181, workflow Autotriage)

Evidence from the log (all times 2026-08-02, UTC)

1. 21:08:41.766 — instrument/job establishes the job's trace and logs it:

   ##[notice]Trace ID: 896c8b337f1f4384ffeeda32a25895a2, Span ID: 310e31cbf9a51834, Trace Deep Link: ...
   

   From here on, every step's env: block shows TRACEPARENT: 00-896c8b337f1f4384ffeeda32a25895a2-310e31cbf9a51834-03 (confirmed through step 24, "Audit pre-agent workspace", at 21:12:58).

2. 21:08:41.839 — the very next step, "Setup Scripts" (github/gh-aw-actions/setup@34802437a...), is invoked with:

   with:
     destination: /home/runner/work/_temp/gh-aw/actions
     job-name: agent
     trace-id: 88bab49c6b45aa6d94f70a47ec31309f
     parent-span-id: 908aa33a2bbf75dc
   

   — a different trace/span than step 1, even though this same step's own env: block still shows TRACEPARENT: 00-896c8b...-310e31c...-03.

3. 21:10:15.998 – 21:10:16.020 — "Setup Scripts"'s own internal logic ([otlp]-prefixed logs) processes those inputs:

   [otlp] INPUT_TRACE_ID=88bab49c6b45aa6d94f70a47ec31309f (will reuse activation trace)
   [otlp] INPUT_PARENT_SPAN_ID=908aa33a2bbf75dc (will parent setup span)
   [otlp] GH_AW_OTLP_ENDPOINTS not set, skipping setup span
   [otlp] resolved trace-id=88bab49c6b45aa6d94f70a47ec31309f
   [otlp] trace-id=88bab49c6b45aa6d94f70a47ec31309f written to GITHUB_OUTPUT
   [otlp] span-id=29a10ff7983cd05d written to GITHUB_OUTPUT
   [otlp] parent-span-id=908aa33a2bbf75dc written to GITHUB_OUTPUT
   [otlp] GITHUB_AW_OTEL_TRACE_ID written to GITHUB_ENV
   [otlp] GITHUB_AW_OTEL_PARENT_SPAN_ID written to GITHUB_ENV
   

   → GITHUB_AW_OTEL_TRACE_ID=88bab49c6b45aa6d94f70a47ec31309f, GITHUB_AW_OTEL_PARENT_SPAN_ID=29a10ff7983cd05d.

4. 21:13:04.818 — step 25, "Execute GitHub Copilot CLI" (the actual agent invocation, and what the built-in observability of the agentic workflow instruments), shows:

   TRACEPARENT: 00-88bab49c6b45aa6d94f70a47ec31309f-29a10ff7983cd05d-01
   

   i.e. the trace from step 3, not the job trace from step 1. Every other step in the job (before and after step 25) keeps using 896c8b....

Net result: two disconnected root traces for one job — 896c8b337f1f4384ffeeda32a25895a2 for ordinary job-level instrumentation, 88bab49c6b45aa6d94f70a47ec31309f for the agent's own execution/observability — rather than a single trace covering the whole job.

Where 88bab49c.../908aa33a... comes from

.github/workflows/autotriage.lock.yml (compiled output — autotriage.md, the hand-authored source, has no OTel/tracing config at all, so this wiring comes from gh aw compile) wires the agent job's Setup Scripts step to the upstream activation job's own setup outputs:

# autotriage.lock.yml, agent job (lines 424, 434-444)
      - uses: plengauer/opentelemetry-github/actions/instrument/job@2169fb0fa9a8260fde99a2a185bbf99a5d63546e # v5.59.0
        ...
      - name: Setup Scripts
        id: setup
        uses: github/gh-aw-actions/setup@34802437a0cba2fd1352d1f1bb02c18a78b56b31 # v0.83.0
        with:
          destination: ${{ runner.temp }}/gh-aw/actions
          job-name: ${{ github.job }}
          trace-id: ${{ needs.activation.outputs.setup-trace-id }}
          parent-span-id: ${{ needs.activation.outputs.setup-parent-span-id || needs.activation.outputs.setup-span-id }}

and, only for the agent-execution step (line 882):

        env:
          ...
          TRACEPARENT: ${{ env.GITHUB_AW_OTEL_TRACE_ID != '' && env.GITHUB_AW_OTEL_PARENT_SPAN_ID != '' && format('00-{0}-{1}-01', env.GITHUB_AW_OTEL_TRACE_ID, env.GITHUB_AW_OTEL_PARENT_SPAN_ID) || '' }}

So 88bab49c.../908aa33a... is what the activation job's own Setup Scripts step produced (job 91547839105, https://github.com/plengauer/Thoth/actions/runs/30767171258/job/91547839105 — that job's gh-aw-actions/setup call also gets no explicit trace-id/parent-span-id, so it self-generates one). It has no relationship to the agent job's own job-level trace (896c8b...) that instrument/job established a fraction of a second earlier, in the same job.

The same pattern (trace-id/parent-span-id sourced from needs.activation.outputs.*, same TRACEPARENT override on the agent-execution step) also appears in the detection job of this workflow (lock file lines ~1062-1077, ~1495), and the same output block shows up in autofix.lock.yml, autoapprove.lock.yml, and monitor_changelog.lock.yml. I have not checked those jobs' own logs, so I can't confirm they show the identical divergence — flagging it only as a plausible wider scope based on the compiled YAML looking the same.

What I could find of "the custom instrumentation"

Looking in plengauer/opentelemetry-github (the job-level instrumentation) for something that would make it accept or reuse an inherited trace-id/parent-span-id instead of always minting a fresh one:

• actions/instrument/job/action.yml declares no trace-id / parent-span-id / similar input at all — only github_token, cache, secrets_to_redact, self_monitoring, self_monitoring_anonymize, debug, and the internal __job_status / __job_matrix / __job_id.
• The one gh-aw-aware piece of custom logic I found is in actions/instrument/job/decorate_action.sh, line 229:

  if [ "$exit_code" = 0 ] && ( [ "${GITHUB_ACTION_REPOSITORY:-}" = github/gh-aw ] || [ "${GITHUB_ACTION_REPOSITORY:-}" = github/gh-aw-actions ] ) && [ "${GITHUB_STEP:-$GITHUB_ACTION}" = setup ] && [ -d "${INPUT_DESTINATION:-}" ]; then
    find "${INPUT_DESTINATION}" -name "*.sh" 2>/dev/null | while IFS= read -r script_file; do
      sed -i 's~#!/bin/sh~#!/bin/sh\n. otel.sh~g' "$script_file" 2>/dev/null || true
      sed -i 's~#!/bin/bash~#!/bin/bash\n. otel.sh~g' "$script_file" 2>/dev/null || true
    done || true
    [ -n "${OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT:-}" ] || echo OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_and_event >> "$GITHUB_ENV"
    echo "::debug::Instrumented agentic workflows"
  fi

  This fires specifically when decorating the gh-aw Setup Scripts step — confirmed for this run, since OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT: span_and_event shows up in every step's env: from "Setup Scripts" onward. But it only (a) sources otel.sh into the *.sh scripts gh-aw copies into place, so later ordinary bash steps get shell-level instrumentation, and (b) sets the GenAI content-capture env var. It doesn't touch trace-id/parent-span-id at all, so this isn't a "reuse the trace" mechanism.
• The actual [otlp] INPUT_TRACE_ID=... (will reuse activation trace) logic lives in github/gh-aw's and github/gh-aw-actions' own actions/setup/js/action_setup_otlp.cjs (upstream, third-party — not plengauer/opentelemetry-github). It just faithfully reuses whatever trace-id/parent-span-id it's handed; in this run that happens to be the activation job's self-generated trace rather than the current job's own trace.

I wasn't able to locate a piece of code in plengauer/opentelemetry-github that fakes/forwards a trace-id/parent-span-id so that a job's own instrument/job trace gets picked up by that same job's gh-aw-actions/setup call. If such a mechanism is supposed to exist, I couldn't find where — flagging that as a gap rather than guessing.

Possibly unrelated

In the same window where instrument/job establishes the trace, its "Start Observation" step also logs:

21:08:41.7277327Z jq: error (at <stdin>:0): null (null) has no keys

Not clear if it's connected to the above; noting it since it's in the same phase of execution.

---

Not attempting to root-cause or fix this here — just documenting what the log and compiled workflow show.

[github-actions]

Triage Analysis

This is a well-documented bug. The root cause is clear:

The agent job in autotriage.lock.yml (and similar compiled lock files) passes needs.activation.outputs.setup-trace-id / setup-parent-span-id to gh-aw-actions/setup, which causes the agentic workflow's built-in observability (GITHUB_AW_OTEL_TRACE_ID / GITHUB_AW_OTEL_PARENT_SPAN_ID) to use the activation job's self-generated trace — completely disconnected from the current job's trace that plengauer/opentelemetry-github/actions/instrument/job establishes.

Potential Fix

The instrument/job action needs to expose its trace-id and span-id as step outputs (or job-level env vars), and the compiled workflow (or the gh aw compile logic) needs to wire those into the gh-aw-actions/setup call's trace-id/parent-span-id inputs instead of using the activation job's outputs.

Specifically, in actions/instrument/job/action.yml, the action should:

1. Output its trace-id and span-id
2. The lock file's Setup Scripts step should reference those outputs: trace-id: ${{ steps.<instrument-job-step-id>.outputs.trace-id }}

Alternatively, the decorate_action.sh hook that already fires when decorating gh-aw-actions/setup (line 229) could be extended to also write the current job trace-id/span-id into the step's trace-id/parent-span-id inputs, though that's trickier since inputs are already resolved.

Scope

As noted in the issue, the same pattern likely affects detection job in autotriage.lock.yml, and autofix.lock.yml, autoapprove.lock.yml, monitor_changelog.lock.yml.

The jq: error (at :0): null (null) has no keys warning is likely unrelated noise from parsing empty/null JSON in the "Start Observation" step.

Assignability

This requires changes to plengauer/opentelemetry-github (actions/instrument/job/action.yml needs new outputs) and possibly the gh aw compile toolchain to wire them correctly. Not a simple single-repo fix — flagging for human review before agent assignment.

@plengauer — could you clarify: is instrument/job already supposed to expose trace/span outputs that the compiled workflow can reference? Or does the fix need to happen inside gh aw compile's wiring logic?

Generated by Autotriage for #3884 · sonnet46 13.6 AIC · ⌖ 5.16 AIC · ⊞ 5.2K · ◷

[plengauer]

it may also be the problem that currently the slim images are not instrumentaiton, and maybe thats the root cause because the trace id is determined there.