Skip to content

feat: report funnel metrics on channel mcp - #52

Draft
Starllordz wants to merge 2 commits into
mainfrom
feat/metrics
Draft

feat: report funnel metrics on channel mcp#52
Starllordz wants to merge 2 commits into
mainfrom
feat/metrics

Conversation

@Starllordz

Copy link
Copy Markdown
Collaborator

Reports funnel metrics to the integrations monitoring backend on channel mcp, so the gap between installs and actual usage can be measured. Both transports report — unlike the private server, which is HTTP-only because its installation id lives in Redis.

Follows the shared event reporting standard (src/modules/metrics/frontend.md in lara-integrations-monitoring).

Events

Event When
install first run only, when the installation id is created. No accountId
auth_success first call that comes back OK — access key credentials are never validated on their own
auth_fail beside a call_error Lara answered with 401/403
call_success / call_error every tool call, with latencyMs and charsTranslated

auth_success is deduplicated per account per process: the HTTP transport is stateless and builds a Translator per request, which would otherwise report one on every tool call.

What is sent

accountId is the Lara account (acc_...) behind the credentials, read from the id claim of the token the SDK holds. An access key is not an account — it belongs to one and is shared by its users — so reporting it would split the same customer across channels. No token means no account and no event: a made-up id is a fake account in every dashboard, which is why a rejected key produces no auth_fail.

metadata carries feature (text / language_detection / resource_management), toolName, transport, and the language tags on translate. Never the translated text, never a credential.

Delivery

Buys a token from /auth/issue-token with the channel API key, posts to /metrics/ingest-events. Batched every 2s behind an unref'd timer, one retry on 401 only, requeued on 429/5xx, dropped on any other rejection, queue capped at 10 000, flushed once on shutdown with a 2s cap.

A backend that is down never delays, alters or fails a translation.

Opt-out

DO_NOT_TRACK set to 1/true/yes and nothing is measured, queued, written to disk or sent. Documented in the README.

The installation id is a random UUID at ${LARA_HOME:-~/.lara}/installation-id, tied to nothing; it exists so repeated runs of the same install count as one. An unwritable home falls back to a per-process id rather than dropping telemetry.

Before this can ship

DEFAULT_METRICS_URL and DEFAULT_METRICS_API_KEY in src/metrics.ts are empty, so telemetry is off for everyone as it stands. They need the deployed backend hostname and a production key issued for the mcp channel — the monitoring backend has no deployment yet, and only ships a development key. METRICS_URL / METRICS_API_KEY override them meanwhile.

Note the key is compiled into a public npm package and will be readable by anyone who installs it. It is write-only, but worth a deliberate decision before filling it in.

Notes

Events are emitted inside CallTool's own try/catch rather than from a wrapper around it: CallTool rewrites LaraApiError into InvalidInputError before it escapes, so only there is the original error still available to classify. A wrapper would have seen every failure as validation_error and auth_fail would never have fired.

The cast through the SDK's protected client moved into src/lara-client.ts, reused by the existing X-Lara-Client headers, so an SDK shape change surfaces in one place.

Testing

60 new tests (226 total, all green): the client in isolation — queue, batching, token cooldown, 401 retry, installation id races, opt-out, payload never carrying credentials or text — and the server wiring end to end.

Verified against a local monitoring backend with real Lara credentials, both transports, reading the rows back out of ClickHouse:

  • stdio: installauth_success → three call_success with the right account, charsTranslated: 19 on a translate whose translatable: false block was correctly excluded
  • http: one auth_success across two stateless requests, then auth_fail + call_error on a revoked key
  • wrong secret → no events at all
  • DO_NOT_TRACK=1, unconfigured defaults, malformed METRICS_URL, unreachable backend → server behaves exactly as before, ~/.lara never created

## Changed
- Tool calls now emit a call_success or call_error, reported from inside
  CallTool's own try/catch so the original SDK error is still available to
  classify before it is rewritten
- Shutdown flushes the queued events before exiting, capped at 2s
- The cast through the SDK's protected `client` moved to a single guarded
  accessor, reused by the existing X-Lara-Client headers

## New
- src/metrics.ts: batched, fire-and-forget event reporting to the
  integrations monitoring backend. Buys a token from /auth/issue-token,
  posts to /metrics/ingest-events every 2s, retries once on 401, requeues
  on 429/5xx, drops on any other rejection, queue capped at 10 000. A
  backend that is down never delays or fails a translation
- Events: install (first run only), auth_success (deduplicated per account
  per process, since the HTTP transport builds a Translator per request),
  call_success, call_error, and auth_fail beside a call_error Lara answered
  with 401/403. Both transports report
- accountId is the Lara account behind the credentials, read from the id
  claim of the token the SDK holds. No token means no event: a made-up id
  would be a fake account in every dashboard
- Installation id persisted at ${LARA_HOME:-~/.lara}/installation-id, with
  a per-process fallback when the home is not writable
- Opt-out via DO_NOT_TRACK: nothing is measured, queued, written to disk or
  sent. METRICS_URL / METRICS_API_KEY override the compiled-in defaults,
  which are empty until the backend is deployed
- 60 tests covering the client in isolation and the server wiring

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are a few correctness/documentation gaps (envelope stamping safety, queue cap enforcement on requeue, and README wording vs. actual “no token → no events” behavior) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds funnel-style usage metrics reporting for the MCP channel, integrating a self-contained telemetry client into both STDIO and HTTP transports without impacting translation behavior when the backend is unavailable or telemetry is disabled.

Changes:

  • Introduces src/metrics.ts to queue, batch, and deliver funnel events (install, auth_success, call_success/call_error, auth_fail) with opt-out and safe-failure semantics.
  • Wires metrics emission into MCP tool execution (CallTool) and session setup, plus flush-on-shutdown handling.
  • Adds env support, documentation, and comprehensive Vitest coverage for client behavior and end-to-end MCP wiring.
File summaries
File Description
src/metrics.ts New telemetry client: installation id, batching, token issuance, error classification, and per-call reporting.
src/mcp/tools.ts Emits call success/error metrics from inside CallTool while preserving existing error rewriting.
src/mcp/server.ts Creates per-session metrics context and centralizes SDK client access via getLaraClient().
src/lara-client.ts Adds a guarded accessor for the SDK’s internal client to centralize the protected cast.
src/index.ts Flushes queued metrics during shutdown with a 2s cap.
src/env.ts Adds metrics-related environment variables and opt-out configuration.
src/tests/server/mcp.metrics.test.ts End-to-end wiring tests for MCP server metrics emission.
src/tests/metrics.test.ts Extensive unit tests for metrics client behavior (queueing, batching, retries, opt-out, etc.).
README.md Documents usage metrics, payload contents, and opt-out behavior.
CLAUDE.md Updates repository guidance to reflect the new metrics module and tests.
.env.example Documents new env vars for metrics configuration and opt-out.
Review details
  • Files reviewed: 11/11 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/metrics.ts Outdated
Comment thread src/metrics.ts Outdated
Comment thread README.md Outdated
## Changed
- The event envelope is stamped last, so channel and sessionId always win
  over whatever a caller passes
- A batch handed back after a failed delivery is trimmed to the queue cap
  immediately; a server going idle while the backend is down used to sit
  above the bound until the next event arrived
- README no longer implies rejected credentials produce an auth_fail: no
  account is known in that case, so no event is sent at all

## New
- Two tests, each verified to fail without its fix: the envelope resists a
  caller overriding it, and a requeued batch stays within the cap
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants