Merge pull request #56 from fangxiu-wf/fix/openclaw-span-lifecycle-and-custom-spanprocessor

feat(openclaw): custom SpanProcessor injection + fix runaway ENTRY/AGENT span durations

Author: fangxiu-wf

opentelemetry-instrumentation-openclaw/CHANGELOG.md

@@ -2,6 +2,32 @@
 
 本文档记录 `opentelemetry-instrumentation-openclaw` 的重要变更。
 
+## [0.1.5-beta] - 2026-07-14
+
+### 新增
+
+- **自定义 SpanProcessor 注入（`spanProcessorModule`）**：
+  - 新增配置项 `spanProcessorModule`（或环境变量 `ARMS_SPAN_PROCESSOR_MODULE`），指向一个默认导出 `SpanProcessor` 的 JS/MJS 模块；绝对路径原样使用，相对路径相对 `OPENCLAW_HOME`（`~/.openclaw`）解析
+  - 用于注入依赖 span 类型/内容的动态属性（如按 model 判定成本档位、按工具名判定工具类别），或将 span 转发到额外后端；补足 `globalSpanAttributes` 仅支持静态值的空缺
+  - 通过包子路径导出 helper：`@loongsuite/opentelemetry-instrumentation-openclaw/span-processor` 提供 `defineGenAiSpanProcessor`，按 GenAI span 类型分派（`onLlmEnding`/`onToolEnding`/`onAgentEnding`/`onStepEnding`/`onEntryEnding`），并屏蔽语义规范方言差异
+  - 加载失败/校验不通过时优雅降级到内置 processor；用户回调全程 try/catch 隔离，绝不影响内置导出管道
+- **`openclaw.trace.close_reason` 诊断属性**：ENTRY span 上标注 trace 关闭方式（`normal` / `runid_recovered` / `stale_sweeper`），便于在 ARMS 中定位异常关闭的调用链
+
+### 修复
+
+- **修复跨天巨长 ENTRY/AGENT span（时长被拉到 1d+）**：
+  - 根因：`agent_end` 仅按 channel 归属 context，当其解析到的 channel 与 `llm_input` 注册的 channel 不一致时，找不到本轮 context、关不掉已开的 ENTRY/AGENT span；这些泄漏的 span 在内存中滞留，直到之后某次 `agent_end` 或会话 reset 时以 `Date.now()` 被误关，产生天级时长
+  - 主修：`agent_end` 在 channel 解析不到可关闭 span 时，用 openclaw 附带的 `runId` 兜回本轮 context（channel 优先、runId 补充，不改动常见路径）
+  - 加固：过期上下文清扫器（sweeper）对仍开放的 ENTRY/AGENT/STEP span 主动强制关闭，使用**有界的最后活动时间**而非 `Date.now()`，并彻底清理所有查找结构
+- **修复工具 span 因 channel 错配被静默丢弃**：`before_tool_call` 在 `agent/` 通道无锚点时，用 `runId` 兜回 context 恢复工具 span（并对 `runId==sessionId` 且上下文正在关闭的场景加守卫，避免挂到上一轮 trace）
+
+### 说明
+
+- 新增单元测试：`span-processor.test.ts`（类型分派/方言容错/优雅降级/运行时隔离）、`channel-mismatch-repro.test.ts`（工具 span channel 错配恢复）、`agent-end-stale-close.test.ts`（`agent_end` runId 兜回 + sweeper 有界强关）
+- `spanProcessorModule` 会加载并执行本地任意代码，仅指向可信模块
+
+---
+
 ## [0.1.4-beta] - 2026-05-26
 
 ### 新增

opentelemetry-instrumentation-openclaw/README.md

@@ -102,6 +102,7 @@ When a config field is not set in `openclaw.json`, the plugin falls back to envi
 | `ARMS_ENABLE_TRACE_PROPAGATION` | `enableTracePropagation` | Enable W3C Trace Context propagation (`true` / `1`) |
 | `OTEL_RESOURCE_ATTRIBUTES` | `resourceAttributes` | Custom resource attributes (`key1=value1,key2=value2`) |
 | `OTEL_SPAN_ATTRIBUTES` | `globalSpanAttributes` | Global span attributes injected to all spans (`key1=value1,key2=value2`) |
+| `ARMS_SPAN_PROCESSOR_MODULE` | `spanProcessorModule` | Path to a module whose default export is a custom SpanProcessor |
 
 Priority: **config file > environment variable > default value**
 
@@ -196,6 +197,87 @@ For `resourceAttributes`, config file values override environment variable value
 
 ---
 
+## Custom SpanProcessor (Dynamic Attributes)
+
+`globalSpanAttributes` only supports **static** values. When you need attributes
+that depend on span type or content (e.g. cost tier by model, tool class by tool
+name), or you want to forward spans to an extra backend, inject a custom
+`SpanProcessor`.
+
+Point the plugin at a module whose **default export** is a `SpanProcessor`:
+
+```json
+{
+  "config": {
+    "endpoint": "https://your-otlp-endpoint:4318",
+    "spanProcessorModule": "./biz-span-processor.mjs"
+  }
+}
+```
+
+- Absolute paths are used as-is; relative paths resolve against `~/.openclaw`.
+- Env fallback: `ARMS_SPAN_PROCESSOR_MODULE`.
+- If the module fails to load or is invalid, the plugin logs an error and keeps
+  working with the built-in processor only (graceful degradation).
+
+### Recommended: the helper API
+
+The package exports a helper that dispatches by GenAI span type and hides both
+the OTel SDK details and the semantic-convention dialect. Write
+`~/.openclaw/biz-span-processor.mjs`:
+
+```js
+import { defineGenAiSpanProcessor } from
+  "@loongsuite/opentelemetry-instrumentation-openclaw/span-processor";
+
+export default defineGenAiSpanProcessor({
+  onLlmEnding(span, { model }) {
+    span.setAttribute("business.cost_tier",
+      model?.includes("gpt-4") ? "premium" : "standard");
+  },
+  onToolEnding(span, { toolName }) {
+    span.setAttribute("business.tool_class",
+      toolName?.startsWith("mcp_") ? "mcp" : "native");
+  },
+  // also available: onAgentEnding / onStepEnding / onEntryEnding
+});
+```
+
+Each hook fires at **`onEnding`**, where the span is still writable and all
+attributes are populated.
+
+### Advanced: a raw SpanProcessor
+
+For full lifecycle control (e.g. forwarding to another backend), export a
+standard `SpanProcessor`:
+
+```js
+export default {
+  onStart(span, parentContext) {},
+  onEnding(span) { span.setAttribute("business.env", process.env.BIZ_ENV ?? "prod"); },
+  onEnd(readableSpan) { /* read-only: observe / forward */ },
+  forceFlush() { return Promise.resolve(); },
+  shutdown() { return Promise.resolve(); },
+};
+```
+
+### Hook points
+
+| Hook | Span writable | Attributes available | Use for |
+|---|---|---|---|
+| `onStart(span)` | Yes | Start-time attributes (incl. `gen_ai.span.kind`); end-time attributes not yet set | Static / env attributes |
+| `onEnding(span)` | Yes | **All attributes** | **Recommended**: dynamic, type-based injection |
+| `onEnd(readableSpan)` | No (read-only) | All attributes | Observation, logging, forwarding |
+
+> Writing attributes in `onEnd` via `readableSpan.attributes[...] = ...` is an
+> unsupported hack that bypasses SDK/plugin truncation and validation. Always
+> write in `onEnding`.
+
+> **Security**: `spanProcessorModule` loads and executes arbitrary local code.
+> Only point it at modules you trust.
+
+---
+
 ## Uninstall
 
 ```bash

opentelemetry-instrumentation-openclaw/openclaw.plugin.json

@@ -1,7 +1,7 @@
 {
   "id": "opentelemetry-instrumentation-openclaw",
   "name": "OpenClaw OTel Plugin",
-  "version": "0.1.4-beta",
+  "version": "0.1.5-beta",
   "description": "Report OpenClaw AI agent execution traces to any OTLP-compatible backend via OpenTelemetry",
   "type": "plugin",
   "entry": "./dist/index.js",
@@ -60,6 +60,10 @@
         "type": "array",
         "items": { "type": "string" },
         "description": "URL substrings for outbound traceparent injection (e.g. [\"api.openai.com\"]). Injects into all HTTPS calls except the OTLP endpoint if omitted."
+      },
+      "spanProcessorModule": {
+        "type": "string",
+        "description": "Path to a JS/MJS module whose default export is a custom SpanProcessor. Absolute paths are used as-is; relative paths resolve against ~/.openclaw. Falls back to ARMS_SPAN_PROCESSOR_MODULE env var. Load failures degrade gracefully to the built-in processor."
       }
     }
   }

opentelemetry-instrumentation-openclaw/package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "@loongsuite/opentelemetry-instrumentation-openclaw",
-  "version": "0.1.4-beta",
+  "version": "0.1.5-beta",
   "description": "OpenTelemetry instrumentation for OpenClaw — report AI agent execution traces to any OTLP-compatible backend",
   "type": "module",
   "license": "Apache-2.0",
@@ -25,6 +25,17 @@
     "loongsuite"
   ],
   "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    },
+    "./span-processor": {
+      "types": "./dist/span-processor.d.ts",
+      "import": "./dist/span-processor.js"
+    }
+  },
   "files": [
     "dist",
     "openclaw.plugin.json"

opentelemetry-instrumentation-openclaw/package.json

@@ -11,12 +11,14 @@ import {
 import {
   BasicTracerProvider,
   BatchSpanProcessor,
+  type SpanProcessor,
 } from "@opentelemetry/sdk-trace-base";
 import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
 import { resourceFromAttributes } from "@opentelemetry/resources";
 import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
 import { hostname } from "node:os";
 import { basename } from "node:path";
+import { loadUserSpanProcessor } from "./span-processor-loader.js";
 import type {
   ArmsTraceConfig,
   OpenClawPluginApi,
@@ -100,9 +102,22 @@ export class ArmsExporter {
       scheduledDelayMillis: this.config.flushIntervalMs,
     });
 
+    // The built-in processor stays first so the export pipeline keeps working
+    // even if a user-provided processor fails to load or throws at runtime.
+    const spanProcessors: SpanProcessor[] = [spanProcessor];
+    if (this.config.spanProcessorModule) {
+      const userProcessor = await loadUserSpanProcessor(
+        this.config.spanProcessorModule,
+        this.api,
+      );
+      if (userProcessor) {
+        spanProcessors.push(userProcessor);
+      }
+    }
+
     this.provider = new BasicTracerProvider({
       resource,
-      spanProcessors: [spanProcessor],
+      spanProcessors,
     });
 
     // Intentionally NOT calling provider.register() to avoid overriding