Feat/otel memory semconv v0.1.0 - #2142
Open
henrikrexed wants to merge 6 commits into
Open
Conversation
Wires the working-draft memory-semconv v0.1.0 conventions into the
MemPalace MCP server. Off by default; activates only when both
OTEL_EXPORTER_OTLP_ENDPOINT is set AND the [observability] extra is
installed. No new runtime deps on the default install path.
Spans
Every MCP tool dispatch emits a memory.<operation> span where
<operation> is one of read | write | invalidate. Attributes:
memory.operation, memory.tool. Argument values are never attached
(PII discipline).
Metrics
search_memories records two histograms per call:
memory_recall_results_count (drawers returned)
memory_recall_top_similarity (cosine sim of top-1 hit)
Resource attributes
memory.sut.name=mempalace, memory.sut.architecture=mcp,
service.name (default mempalace-mcp, OTEL_SERVICE_NAME override),
service.version (from mempalace.version).
Files
+ mempalace/telemetry.py (new — lazy init, no-op when disabled)
+ docs/observability.md (new — opt-in setup + cardinality notes)
+ docs/verification/mempalace-baseline.dql (new — Dynatrace queries)
M mempalace/mcp_server.py (init at startup + span around dispatch)
M mempalace/searcher.py (record_recall after re-rank)
M pyproject.toml ([observability] extra; OTel API/SDK/OTLP)
Verified
* uv sync --extra dev --extra observability — clean install
* 193 tests pass (test_mcp_server + test_searcher + test_mcp_stdio_protection)
* ruff check — clean
* No-op path: env unset → init_telemetry returns False, dispatch
yields _NOOP_SPAN, record_recall short-circuits
* Live path: env set + collector unreachable → spans buffer, retries
log warnings, server keeps serving (telemetry never blocks dispatch)
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the opt-in OTel integration so it now covers all three pillars
(traces, metrics, logs) and accepts trace context from MCP clients via
params._meta. An agent + MemPalace now share a single trace_id when
the client injects the W3C tracecontext headers.
Logs
init_telemetry now also configures a LoggerProvider + OTLPLogExporter
and bridges stdlib `logging` -> OTel via LoggingHandler attached to
the `mempalace` logger (not root, so chromadb/posthog keep their
existing stderr-only behavior). Every dispatch emits one structured
record:
"memory.dispatch tool=<tool_name> operation=<read|write|invalidate>"
Records carry trace_id + span_id of the active span.
Trace context propagation
New extract_trace_context(meta) parses W3C tracecontext headers
from params._meta using TraceContextTextMapPropagator. memory_operation
now accepts a parent_context parameter and starts the memory.<op>
span as a child of the remote parent when supplied.
The MCP dispatch wrapper in mcp_server.py extracts params._meta and
passes it through, so any client that already owns a trace can fan
into MemPalace without code changes.
Sentinel object distinguishes "no parent" from "explicit None" so
callers can pass either without ambiguity. Malformed headers fall
back to a new trace and log at debug -- never raise.
Coverage
+ tests/test_telemetry.py (7 tests, no-op path)
+ 2 new DQL queries in docs/verification/mempalace-baseline.dql:
Q7 logs with trace correlation by operation/tool
Q8 end-to-end traces with multi-service fan-out
M docs/observability.md -- logs section + reference client snippet
that shows how to inject tracecontext into params._meta
Verified
* full test suite: 190 mcp+searcher tests still pass, 7 new
telemetry tests pass, ruff clean
* Live tenant verification against dynatrace-dev:
- 4 trace_ids each contain both `verify-agent` and
`mempalace-mcp-e2e` services under one id (end-to-end
propagation confirmed)
- 4 `memory.dispatch tool=... operation=verify` log records
landed in the logs pillar
* No-op path: all new paths short-circuit when env unset
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ace loggers
Two issues caught by end-to-end testing against a real mempalace-mcp
server (the in-process smoke from the previous commit didn't surface
either):
1. The structured `memory.dispatch` log line was emitted unconditionally
from the dispatch wrapper. On the default install path (no
OTEL_EXPORTER_OTLP_ENDPOINT) it leaked one INFO line per call to
stderr -- noise with no consumer. Guard with `is_enabled()` so the
line only fires when telemetry is on and there's somewhere for it
to go.
2. The OTel LoggingHandler was only attached to the `mempalace` logger
(dotted). But mcp_server.py / searcher.py / palace.py / miner.py /
fact_checker.py / query_sanitizer.py / convo_miner.py all use the
underscore name `mempalace_mcp`, which is NOT a child of the dotted
tree. Result: the dispatch log line landed at OTel logger setup,
the embedding init line landed, but none of the per-call dispatch
logs reached OTLP. Attach the handler to all five MemPalace-owned
logger names (`mempalace`, `mempalace_mcp`, `mempalace_graph`,
`mempalace_hallways`, `mempalace_format_miner`).
Verified
* tests/test_mcp_server.py + tests/test_telemetry.py: 167 pass
* full suite: 2087 passed, 3 skipped
* ruff clean
* /tmp/e2e_mcp_optout.py drives a real mempalace-mcp subprocess with
no OTel env -- stderr now free of `memory.dispatch` markers; module
state confirms no LoggingHandler attached
* /tmp/e2e_mcp_stdio.py drives a real mempalace-mcp subprocess with
OTel pointed at dynatrace-dev. Confirmed in DT:
- 4 spans: memory.read(3) + memory.write(1) keyed by memory.tool
- 4 log lines: `memory.dispatch tool=X operation=Y` per call
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The upstream fork sync (ISI-1920) added mempalace_checkpoint,
delete_by_source, delete_hallway, mine, list_hallways, and
kg_supersede. These were falling through operation_for_tool()'s
default ('read'), mislabeling four mutating tools and one fact
retraction. Classify them explicitly: checkpoint/delete_by_source/
delete_hallway/mine -> write, list_hallways -> read, kg_supersede
-> invalidate (atomic retire+write at a boundary, like kg_invalidate).
Also regenerates uv.lock for the [observability] extra against the
freshly synced dependency set.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tream PR Remove internal issue-tracker/project references (ISI-1068, ISI-1920, "kagent") from the instrumentation docs and code comments ahead of the upstream contribution, and format the telemetry dispatch lines to the project's line-length=100 so `ruff format --check` passes in CI. No behavior change: comment/whitespace only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve conflict in mempalace/mcp_server.py: nest develop's new _write_stall_watch watchdog inside this branch's memory_operation tracing span so both wrap the handler call (span covers the stall watch). Adopts develop's ruff 0.16.1 pin — lint gates (ruff check / ruff format --check) pass under 0.16.1; pin-match + telemetry + stall-watch tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>
igorls
added
enhancement
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds opt-in OpenTelemetry instrumentation to
mempalace-mcp, emitting allthree signal pillars — traces, metrics, and logs — on the core memory
operations, mapped to the
memory-semconv v0.1.0working draft.It is off by default and zero-overhead when off: unless
OTEL_EXPORTER_OTLP_ENDPOINTis set and the[observability]extra isinstalled,
telemetry.pyimports no SDK, allocates nothing, and every entrypoint is a no-op. The default install path is unaffected.
What it emits when enabled:
memory.<op>span (read/write/invalidate) aroundevery MCP tool call, via a taxonomy mapping each tool to its semconv
operation kind. Resource attributes:
memory.sut.name=mempalace,memory.sut.architecture=mcp,service.name,service.version.memory_recall_results_countandmemory_recall_top_similarity(histograms) recorded on search.LoggingHandlerbridges MemPalace's stdlib loggers to OTLP sorecords emitted inside a span carry its
trace_id/span_idfor log↔tracecorrelation. Attached only to MemPalace-owned logger names, keeping the MCP
stdio protocol clean.
traceparent/tracestatefrom eachtools/callrequest's_meta, so thememory.<op>span nests under the caller's trace: one end-to-end traceacross the agent's MCP client and MemPalace's handling.
PII / cardinality: raw memory content is never attached to spans — only the
tool name and operation kind. Recall is captured as counts and top-similarity,
not query text.
Files:
mempalace/telemetry.py(new) — the whole feature behind the opt-in gate.mempalace/mcp_server.py— wraps dispatch in the span + extracts inboundtrace context (no-op when off).
mempalace/searcher.py— records recall metrics.pyproject.toml— new optionalobservabilityextra (OTel API/SDK + OTLPHTTP exporter); not a core dependency.
docs/observability.md— enablement, env vars, backend examples, fullspan/metric/log reference.
docs/verification/mempalace-baseline.dql— copy-paste verification queries.tests/test_telemetry.py(new) — taxonomy, no-op gate, span emission,context extraction, recall metrics.
License: MIT (matches the project).
How to test
Drive a few tool calls, then confirm
memory.read/memory.write/memory.invalidatespans — plus the two recall metrics and correlated logs —land in your OTLP backend (Dynatrace, Grafana Tempo/Mimir, Honeycomb, Jaeger,
…).
docs/verification/mempalace-baseline.dqlhas ready-made queries thatassert the emitted signals match the semconv contract.
Checklist
python -m pytest tests/ -v)ruff check .)