Skip to content

feat(openclaw): custom SpanProcessor injection + fix runaway ENTRY/AGENT span durations#56

Merged
fangxiu-wf merged 3 commits into
alibaba:mainfrom
fangxiu-wf:fix/openclaw-span-lifecycle-and-custom-spanprocessor
Jul 14, 2026
Merged

feat(openclaw): custom SpanProcessor injection + fix runaway ENTRY/AGENT span durations#56
fangxiu-wf merged 3 commits into
alibaba:mainfrom
fangxiu-wf:fix/openclaw-span-lifecycle-and-custom-spanprocessor

Conversation

@fangxiu-wf

@fangxiu-wf fangxiu-wf commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Closes #54

Summary

Two changes to opentelemetry-instrumentation-openclaw, released together as 0.1.5-beta.

Feature — custom SpanProcessor injection

  • New config spanProcessorModule (env fallback ARMS_SPAN_PROCESSOR_MODULE): points at a module whose default export is a SpanProcessor. Absolute paths as-is; relative resolved against OPENCLAW_HOME (~/.openclaw).
  • Package subpath export @loongsuite/opentelemetry-instrumentation-openclaw/span-processor provides defineGenAiSpanProcessor, dispatching by GenAI span kind (onLlmEnding/onToolEnding/onAgentEnding/onStepEnding/onEntryEnding) and hiding semconv dialect differences.
  • Graceful degradation: load/validation failure falls back to the built-in processor; every user callback is try/catch-isolated so a faulty processor can never break the export pipeline.
  • Fills the gap that globalSpanAttributes only supports static values.

Fix — runaway ENTRY/AGENT span durations (seen as 1d+ traces)

  • Root cause: agent_end attributed context purely by channel. When its channel differs from the one llm_input registered (system/ vs agent/ mismatch), it failed to find the turn's context and closed nothing, leaking the open ENTRY/AGENT spans. A later agent_end (or session reset) then mis-closed them with Date.now(), producing day-long durations.
  • Primary fix: when channel resolution yields no closable span, recover the turn's context via the runId openclaw attaches to agent_end (channel-first, runId as supplement — the common path is unchanged).
  • Hardening: the stale-context sweeper now force-closes still-open ENTRY/AGENT/STEP spans using a bounded last-activity time (never Date.now()) and fully cleans up all lookup structures.
  • Also fixes tool spans silently dropped on the same channel mismatch (before_tool_call recovers via runId, with a guard for the runId==sessionId closing-turn case).
  • New diagnostic attribute on ENTRY: openclaw.trace.close_reason ∈ {normal, runid_recovered, stale_sweeper}.

Test plan

  • tsc --noEmit clean
  • Unit tests 143/143 pass — new: span-processor.test.ts (dispatch / dialect tolerance / graceful degradation / runtime isolation), channel-mismatch-repro.test.ts (tool-span recovery), agent-end-stale-close.test.ts (agent_end runId recovery + sweeper bounded force-close)
  • Real ARMS link verified: healthy traces report close_reason=normal with sane second-level ENTRY/AGENT durations and intact hierarchy (no regression); a custom SpanProcessor was injected and confirmed to add business.* attributes onto LLM/ENTRY/TOOL spans in ARMS.

🤖 Generated with Claude Code

fangxiu-wf and others added 3 commits July 14, 2026 10:13
…ENT span durations

新增:
- 自定义 SpanProcessor 注入:配置项 spanProcessorModule / 环境变量 ARMS_SPAN_PROCESSOR_MODULE。
  通过包子路径导出 defineGenAiSpanProcessor helper,按 GenAI span 类型分派并屏蔽语义规范方言差异;
  加载/校验失败优雅降级到内置 processor,用户回调全程 try/catch 隔离,绝不影响内置导出管道。
- ENTRY span 诊断属性 openclaw.trace.close_reason(normal / runid_recovered / stale_sweeper),
  便于在 ARMS 中定位异常关闭的调用链。

修复:
- 跨天巨长 ENTRY/AGENT span(时长被拉到 1d+):agent_end 仅按 channel 归属 context,当其 channel
  与 llm_input 注册的 channel 不一致时关不掉本轮 span 导致泄漏,后续某次 agent_end 或会话 reset
  时以 Date.now() 误关产生天级时长。主修:channel 解析不到可关 span 时用 openclaw 附带的 runId
  兜回本轮 context(channel 优先、runId 补充,不改动常见路径)。加固:过期上下文清扫器对仍开放的
  ENTRY/AGENT/STEP span 用有界最后活动时间强制关闭,并彻底清理所有查找结构。
- 工具 span 因 channel 错配被静默丢弃:before_tool_call 在 agent/ 通道无锚点时用 runId 兜回
  context 恢复工具 span,并对 runId==sessionId 且上下文正在关闭的场景加守卫,避免挂到上一轮 trace。

版本 bump 0.1.5-beta;新增 span-processor / channel-mismatch-repro / agent-end-stale-close 单元测试。

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously the sweeper's forceFlush error was silently swallowed, hiding
export-pipeline faults. Log it at warn level when debug is enabled.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@fangxiu-wf fangxiu-wf merged commit 60c560a into alibaba:main Jul 14, 2026
7 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support custom SpanProcessor injection for openclaw instrumentation plugin

1 participant