Skip to content

Commit 135d63c

Browse files
cteytonclaude
andauthored
chore(observability): add grafana/otel-lgtm to the local dev stack (#431)
* chore(observability): add grafana/otel-lgtm to the local dev stack Adds a self-hosted OTLP backend (OTel Collector + Tempo + Prometheus + Loki + Grafana in one container) for local trace analysis, behind an `observability` compose profile so it does not weigh on the default dev stack. Tracing is off unless OTEL_EXPORTER_OTLP_ENDPOINT / VITE_OTEL_EXPORTER_URL are set, which keeps the exporter from retrying against a host that is not running. No custom collector config is mounted: the image already enables CORS on the OTLP/HTTP receiver, which is all the browser exporter needs. The image tag is pinned because its Grafana provisioning and collector config are internal details that change between releases. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * feat(observability): trace the API with OpenTelemetry down to pg queries Adds apps/api/src/otel.ts, starting the OTel NodeSDK with the auto-instrumentation bundle and exporting OTLP. Gives request -> Nest controller -> SQL statement spans, plus Redis and outgoing LLM calls (the OpenAI/Anthropic/Google SDKs use global fetch, which the http instrumentation does not patch - undici does). The SDK starts synchronously at module top level and otel.ts is the first import of instrument.ts. Both matter: auto-instrumentation patches modules as they are required, so anything loaded first is never traced. That is also why otel.ts imports no @packmind/* module - @packmind/logger pulls in winston, and a winston required ahead of the hooks would lose log/trace correlation. Verified on the built bundle: the [otel] line prints before any application log, and Winston records then carry trace_id/span_id. Everything is gated on OTEL_EXPORTER_OTLP_ENDPOINT, which is unset in production and in tests, so the SDK never starts there. pg keeps enhancedDatabaseReporting off, so spans carry parameterized SQL but never bind values, and GenAI message content is left uncaptured - neither user data nor prompts reach the trace backend. OTel packages are added to apps/api/docker-package.json as well, since that file is hand-maintained and a missing entry breaks the production container. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * feat(logger): surface trace ids in console log output @opentelemetry/instrumentation-winston injects trace_id, span_id and trace_flags into every record logged inside an active span, so correlation needs no code here. But those three fields landed in the JSON metadata blob of every single console line, drowning the actual payload. Renders a short [trace=xxxxxxxx] marker instead and keeps the three fields out of the blob. The json() format still carries them in full, which is what Loki and Grafana's trace<->logs navigation use. Extracts the printf callback into an exported formatConsoleLine so it can be tested directly: the console transport is unreachable under Jest, which sets PACKMIND_LOG_LEVEL=silent globally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * feat(frontend): emit browser traces over OTLP Adds OtelService with a WebTracerProvider exporting OTLP, so a page interaction and the API work it triggers land on a single trace. Both XHR and fetch are instrumented: Axios uses the XHR adapter in the browser, so XHR is what covers packmindApiService, while fetch covers React Router's own requests. Initialization lives in entry.client.tsx rather than root.tsx, where initSentry sits: React Router runs in SPA mode but still prerenders the shell at build time, so root.tsx module scope executes in Node, where window and XMLHttpRequest do not exist. Confirmed on a real build - the prerender logs Sentry's message but not this one, and the OTel code lands only in the entry.client chunk. No CORS or propagateTraceHeaderCorsUrls config is needed: the API is reached through the relative /api path, so requests are same-origin and traceparent is attached by default. Only the exporter POST is cross-origin, which the collector already allows. Gated on VITE_OTEL_EXPORTER_URL, unset by default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * fix(observability): flush OpenTelemetry before the process exits Review feedback on #431: the SIGTERM/SIGINT listeners in otel.ts started sdk.shutdown() without awaiting it, while main.ts's own listener ended its sequence with process.exit - which terminates regardless of an in-flight export. Reproduced against the built bundle with a stub OTLP collector, real postgres and redis: sending SIGTERM inside the BatchSpanProcessor window produced zero exports, and a control run confirmed spans were being generated (a 22KB batch exported once the 5s window elapsed). So it was not a partial race - the entire final batch was lost, every time. Replaces the listeners with an exported shutdownOtel() that main.ts awaits as the last step before process.exit, in both the success and failure paths. Two listeners racing each other cannot be made correct; shutdown ordering belongs where it already lives. The flush is bounded by a 2s timeout so an unreachable collector cannot hold the process past a container's SIGKILL grace, and it never rethrows, so a failed flush cannot change the exit path. Verified after the change: trace export lands 26ms after SIGTERM and the process exits at 205ms; with the collector unreachable, shutdown completes in 2.2s instead of hanging on exporter retries. Also documents the Node runtime metrics the SDK exports by default, which the collector stub revealed and the README did not mention. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * chore(observability): bind otel-lgtm ports to loopback Review feedback on #431. OTLP ingestion is unauthenticated and Grafana ships on default credentials, while traces carry SQL statements, request URLs and log lines - so publishing on every interface exposed all of it to the local network. Functionally free: the browser reaches the collector over loopback, and the backend reaches it as http://otel-lgtm:4318 over the compose network, which host port bindings do not affect. Confirmed via `docker compose config` that all three now resolve to host_ip 127.0.0.1. Documents the choice in both the compose file and the README, since the ports will look dead to anyone driving Docker from another machine. Note this does not make the dev stack safe on a hostile network: postgres, redis and pgadmin are still published on 0.0.0.0 with weak or absent credentials. Hardening those is a wider change than this PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * ci(observability): inject browser tracing endpoint for Cloud builds only VITE_OTEL_EXPORTER_URL is baked into the client bundle by Vite and cannot be changed after the image is built, so the Cloud/self-hosted split has to happen at build time. Gates it on the same expression the Sentry and Crisp values already use - proprietary edition AND not a release/* tag - since every release/* tag produces the self-hosted images. Adds a guard that fails the frontend build if any Cloud-only VITE_ value is non-empty on a release/* tag. The failure it prevents is invisible: a leaked endpoint in a customer's bundle would silently point their browsers at Packmind infrastructure, and nothing in the running product would look wrong. It covers the Sentry and Crisp values too, which had no such check. Verified both directions by building the frontend with a sentinel URL: the value reaches the bundle when set, and with the empty value the gate produces the bundle contains VITE_OTEL_EXPORTER_URL:"" and initOtel() returns early. Worth checking because getEnvVar reads import.meta.env with a dynamic key, which Vite can only satisfy by serializing the whole env object. The API needs no equivalent change - OTEL_EXPORTER_OTLP_ENDPOINT is read at runtime, so the image is neutral and self-hosted deployments never start the SDK. Documents that in the self-hosted compose file, and the whole split in docker/otel/README.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * docs(observability): correct the pg span attribute name to db.query.text Captured real spans against a migrated postgres to check what the pg instrumentation actually records, and the SQL is not under `db.statement` as documented - instrumentation-pg has moved to the stable database semantic conventions, so it is `db.query.text`, alongside `db.system.name` and `db.namespace`. Anyone following the old name would have concluded the SQL was not captured at all. The substance was right: the full TypeORM-generated statement is recorded, and bind values are not. Confirmed by calling check-email-availability with a real address and finding only `LOWER("user"."email") = LOWER($1)` in the span, with the address nowhere in it. Documents that example, since "parameterized" is easier to trust when you can see what it looks like. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * fix(observability): cut a request from 24 spans to 6 Captured a real trace for one endpoint call and it was close to unreadable: 24 spans, of which 18 were Express middleware and routing noise, mostly named "middleware - patched", burying the four spans that carry the story. Two causes: - Per-middleware spans (cookieParser, jsonParser, cors...) each wrapped in an uninformative parent. Dropped via ignoreLayersType. - Express 5 routes through the standalone `router` package, which the auto bundle instruments separately - so routing was traced twice, contributing 9 opaque spans plus a duplicate of the request-handler span express already emits. Disabled; express covers the same ground. The same request now produces 6 spans reading HTTP -> route -> controller -> use case -> SQL, which is the shape someone new to tracing can actually follow. Declares @opentelemetry/instrumentation-express directly rather than relying on it transitively, since pnpm's isolated node_modules would not resolve it at runtime, and adds it to docker-package.json for the production image. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * fix(observability): actually export logs, so trace correlation works Verified the correlation end to end by capturing traces and logs from one request and cross-referencing, and found that no log records were being exported at all. The trace_id visible in console output came from the injection half of instrumentation-winston, which made it look like everything worked. Cause: log sending needs @opentelemetry/winston-transport, which is an OPTIONAL peer of instrumentation-winston. Without it the instrumentation sends nothing and only emits an OTel diag warning - invisible unless OTEL_LOG_LEVEL is set. So Loki would have stayed empty, and the Tempo<->Loki click-through that justified enabling log sending in the first place would not have existed. Adds the package to the root dependencies and to docker-package.json, and warns against pruning it in both the code comment and the README. Confirmed after the fix, on a single request: - 5 log records carry the same trace_id as the HTTP span, with span ids belonging to spans in that trace - startup logs correctly carry no trace context - an inbound W3C traceparent is adopted: the server span takes the caller's trace id and parent span id, and the request's log records carry the caller's trace id too - which is the browser -> API -> SQL -> logs path Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * docs(observability): document TraceQL duration queries Adds the "show me everything slower than X" entry point, and the distinction that catches people out: duration filters a single span, traceDuration filters the whole request, so {duration > 800ms} finds a slow query even inside a fast request. Verified against a real Tempo 3.0.3 and Loki 3.7.6 running locally, fed by the built API through a splitter standing in for the collector. Looking up one trace id returned its 5 log lines from Loki and all 6 spans from Tempo, and the duration filters selected exactly the slow traces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * docs(observability): document latency percentiles per operation Answers "distribution time with percentiles for each unique method": Tempo's metrics-generator emits traces_spanmetrics_latency_bucket labelled by span_name, so percentiles per endpoint need no extra instrumentation - only a histogram_quantile query. Verified end to end rather than described, by running Tempo 3.0.3 with the span-metrics processor remote-writing to Prometheus 3.13.2 and driving the built API with mixed fast and lock-delayed traffic. Measured, from that run: span p50 p95 p99 POST /../check-email-availability 3.1ms 46.4ms 216.3ms pg.query:SELECT packmind 1.1ms 21.6ms 213.8ms which is also the example the docs use, since the matching p99s show the tail sitting in the query rather than in our code. Records two gotchas found while testing: rate() with no traffic in the window returns NaN, so a quiet environment looks broken when it is not, and unmatched routes collapse into a single bare "POST" span name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * feat(observability): ship a ready-made Grafana dashboard Writing PromQL and TraceQL by hand for every question is too much friction for day-to-day use, so the queries are now written once and provisioned. Adds docker/otel/grafana/dashboards/packmind-api.json - latency percentiles per endpoint, request and error rates, a latency heatmap with an endpoint selector, and database call percentiles - plus the provisioning file and the compose mounts. The otel-lgtm image provisions datasources but ships no dashboards, so this fills the gap; datasource uids (prometheus/tempo/loki) are stable in the image, so the panels bind without manual selection. Also documents that the image already installs the Drilldown apps (exploretraces, lokiexplore, metricsdrilldown), which are point-and-click and cover most exploration with no query language at all. The README now leads with those and frames the query sections as the escape hatch rather than the entry point. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * docs(observability): warn about span-metric naming fragmentation Importing a community dashboard from grafana.com is the obvious reflex, and it will usually show empty panels. Span-metric names differ across the ecosystem: this stack's Tempo metrics-generator emits traces_spanmetrics_latency_bucket (confirmed by querying Prometheus directly), while the Collector's spanmetrics connector and Alloy emit traces_spanmetrics_duration_milliseconds_bucket, and older Tempo used traces_spanmetrics_duration_seconds_bucket. Documents the mismatch and the two ways out, so an empty panel does not get read as a broken pipeline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * docs(observability): make the dashboard portable and document the Cloud move "Just change the endpoint" covers the API and nothing else, so this writes down what actually has to happen. Verified: the SDK reads OTEL_EXPORTER_OTLP_HEADERS natively, so authenticating to Grafana Cloud needs no code change. Confirmed by pointing the built API at an endpoint requiring auth and observing Authorization: Basic on /v1/traces and /v1/logs. Not covered by an endpoint change, and now documented: - browser traces cannot go direct, since VITE_OTEL_EXPORTER_URL is resolved by the browser and would expose the Cloud token in a public bundle - span metrics are off by default in Cloud and billed as active series, so the RED panels stay empty until enabled - nothing is sampled and every log line is exported, which is fine on a laptop and expensive in production Also replaces the dashboard's hardcoded datasource uid with a ${ds} variable, so the same JSON works against otel-lgtm and against Cloud, where uids differ. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * revert(observability): drop browser tracing, keep it to the API Removes the frontend side of the OpenTelemetry work: OtelService, its call in entry.client.tsx, the four browser-only packages, VITE_OTEL_EXPORTER_URL in compose and in both workflows, and the matching documentation. Sending telemetry from a page has no good answer here. The endpoint has to be reachable by the browser, so it either ships a credential in a public bundle or becomes an unauthenticated ingest path on our own domain - and fetching the credential from an API changes nothing, since anything the page can read, anyone can. Neither trade was worth what browser spans add on top of server-side tracing. Keeping it server-side also removes a whole class of problem: the API reads its endpoint at runtime, so the image is neutral and no build-time gate is needed to keep telemetry out of self-hosted images. The CI guard stays for the Sentry and Crisp values, which are still baked into the bundle and had no such check before. Inbound traceparent handling is untouched, so an instrumented caller can still join the trace if browser tracing is ever revisited. Verified: frontend typecheck, lint, 1070 tests and a from-scratch build all pass, and the rebuilt client bundle contains no OpenTelemetry code. API typecheck and build unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * feat(observability): require an explicit environment before exporting deployment.environment.name used to fall back to NODE_ENV, which the API image hardcodes to production. Staging would therefore have announced itself as production, merging its traces, logs and latency percentiles into the production ones with nothing looking wrong. The environment now has to be declared in OTEL_RESOURCE_ATTRIBUTES, which is where the SDK's own resource detector reads it, so there is a single source of truth and no default that can silently win. With an endpoint set and no environment, the SDK does not start and logs an error. The API still boots and serves traffic in that case: a telemetry misconfiguration must not take the service down, while mislabelled telemetry is worse than none - hence loud, but never fatal. Verified on the built bundle: - endpoint set, environment missing -> API up, error logged, collector receives nothing - endpoint + environment=staging -> exports with deployment.environment.name=staging - no OTEL_* variables at all -> API up, no output, no spurious error Compose defaults the local value so local remains a one-variable opt-in, and the README gains the per-environment table. It also records that the attribute does not reach span metrics, so two environments sharing one Prometheus would blend their percentiles. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * docs(observability): correct the multi-environment guidance The previous wording said the environment attribute cannot reach span metrics and implied separate stacks were the only reliable answer. Grafana Cloud's metrics-generator can promote span and resource attributes to dimensions, so deployment.environment.name can become a label on traces_spanmetrics_* on a single stack - with the caveat that each dimension multiplies active series and is billed. Documents both options and when each is the right trade, plus the multi-stack datasource that queries several stacks at once. Notes that the bundled dashboard's ${ds} variable is what makes one JSON serve both environments. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * docs(observability): rule out fetching OTel config at runtime, fix sampling advice Two corrections to guidance, both in areas someone would reasonably get wrong. Configuration.getConfig() must not be used for the OTel bootstrap. It is async and may call Infisical, so the SDK would start only once that promise resolved - after pg, ioredis, express and winston were required, and therefore never patched. Nothing crashes, there are simply no spans. The Sentry init in instrument.ts already demonstrates the trap, its DSN resolving after bootstrap has begun. Records that these have to be real environment variables, and that the only sensitive one can come through the entrypoint mechanism the container already uses for its other secrets. Replaces the "add traceidratio 0.1" advice, which was too blunt: head sampling drops 90% of errors and slow requests along with everything else, and Tempo's metrics-generator only sees what arrives, so percentiles end up computed on the sample. Documents tail sampling in a collector instead - 100% in, decide once the trace is complete, metrics computed before the sampling stage. Backs it with a measurement: 50 real requests on the simplest endpoint cost 6.4 KB of traces and 7.4 KB of logs each, uncompressed. Logs are the bigger half, and trace sampling does nothing about them - so filtering logs buys more than sampling does, which reverses the priority previously given. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * revert(ci): restore build.yml, its guard no longer belongs here The frontend build guard was added to keep the browser OTLP endpoint out of self-hosted bundles. That variable is gone, so what remained only protected the Sentry and Crisp values - which were already gated by the same expressions before this branch existed, and which this PR was never about. build.yml is now byte-identical to main, and main.yml already was. The CI diff of this PR is empty. The gap it covered is real: nothing verifies that the gate expressions actually resolved to empty on a release/* tag, and a mistake there is invisible once bundled. That deserves its own change rather than riding along with API tracing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s * feat(observability): add withSpan, the first manual span helper Auto-instrumentation only patches known library modules - http, express, nestjs-core, pg, winston - so everything in packages/* is invisible in a trace. A waterfall drops from the Nest handler span straight to pg.query with the services, adapters and use cases in between unaccounted for. withSpan() closes that gap. @opentelemetry/api was already a declared dependency imported nowhere; this is its first caller. It degrades cleanly: with no SDK registered - unit tests, or the API started without OTEL_EXPORTER_OTLP_ENDPOINT - trace.getTracer() returns a no-op tracer, the callback still runs, and the cost is a function call. Not usable from apps/api/src/otel.ts: the bootstrap must not import @packmind/*, or winston loads before sdk.start() and log/trace correlation breaks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(observability): give every use case a span Between the Nest handler span and the pg spans the waterfall was blank - services, hexa adapters and use cases all invisible - so a slow request told you it was slow without telling you where. AbstractMemberUseCase.execute() is the single chokepoint for every authenticated use case: AbstractAdminUseCase, AbstractSpaceMemberUseCase and AbstractSpaceAdminUseCase all extend it and inherit this method. One withSpan() here names the whole domain layer, forever, with no per-use-case opt-in. The span carries no attributes on purpose. userId and organizationId would put identifiers in the trace backend, the same reason pg keeps enhancedDatabaseReporting: false. The class name is the payload. Costs roughly one span per request against the six documented in docker/otel/README.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(observability): add a demo slow span to the space skills list DEMO ONLY - revert this commit once the Grafana screenshots are taken. Gives the OTLP setup something unmistakable to look at: a span named thisMethodTakesTwoSeconds, on GET /organizations/:orgId/spaces/:spaceId/skills, found with `{ name = "thisMethodTakesTwoSeconds" }` in TraceQL and nested under the ListSkillsBySpaceUseCase span from the previous commit. The wait is awaited, not a busy-wait: the event loop stays free, so concurrent requests are unaffected while the span still reads exactly like a genuinely slow call. Do not let this reach a deployed environment - it is an unconditional two second regression on the skills list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(observability): document manual spans, fix a mislabelled one The waterfall in section 1 labelled the second Nest row "use case". It is not: instrumentation-nestjs-core emits a REQUEST_CONTEXT span named Controller.method and a REQUEST_HANDLER span named after handler.name alone, so both rows are the controller. It patches RouterExplorer.createHandler and nothing else, which means a bare method name in a waterfall is still the route handler - and nothing from packages/* was in that trace at all. That mislabel is worth correcting because it invites exactly the wrong conclusion: that the domain layer is already traced and a slow use case would show up on its own. Adds a section on withSpan() and the automatic use-case span, with a real capture of the space skills endpoint. Two things worth knowing that are easy to get wrong: the TraceQL intrinsic for instrumentation scope is `instrumentation:name` and `scope.name` does not parse, and spans deliberately carry no attributes so identifiers stay out of the backend. Also drops "Express middleware" from the instrumented table - those spans have been deliberately off since ae2f87b. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(observability): record @opentelemetry/api in the lockfile Follows the dependency added to packages/node-utils for withSpan. The eslint `deprecated` line comes along because pnpm renormalised the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(observability): make the skills SQL slow too, for the demo DEMO ONLY - revert with the rest of the span demo. thisMethodTakesTwoSeconds proves a first-party span shows up, but says nothing about slow SQL. This puts the delay in the real skills SELECT instead of a synthetic sleep statement, so the pg.query span is the actual query with the sleep visible in db.query.text - what a genuinely slow statement looks like. Bind values stay parameterized, so nothing about the caller leaks. The subquery form is load-bearing. `(SELECT pg_sleep(2)) IS NOT NULL` is uncorrelated, so Postgres hoists it to an InitPlan and runs it once - measured at 1.0s across 5 rows. Bare as `pg_sleep(2) IS NOT NULL` it is volatile and runs per row: 4 rows at 0.5s each took 2.0s. IS NOT NULL on void is true, so no rows are filtered. Needs the pg_sleep stub in test-utils because the repository specs run on pg-mem, which implements very few native functions and fails the query outright without one. The stub does not sleep, so the suite stays fast. Note the blast radius: findBySpaceId also backs skill create, update and upload duplicate-name checks and two dashboard use cases, so those pay the two seconds as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(observability): tag traces with the organization id Filtering traces per customer meant reading URLs by eye. Now `{ span.packmind.organization.id = "..." }` does it. Set in two places, deliberately. instrumentation-http's startIncomingSpanHook puts it on the ROOT span, which is what Tempo's trace list and the Drilldown filters read - an attribute buried on a child span is findable in TraceQL but leaves the trace list unfilterable. That hook runs at span creation, before auth, so it has only the URL to go on and matches a strict UUID shape, keeping a literal ":orgId" or a slug out of the attribute. AbstractMemberUseCase then sets the same attribute from the validated command, before validation rather than after, so a rejected request is still attributable to whoever made it. This reverses the "no attributes" call from 568380a, on the evidence that organizationId is already a Loki label on every log line the winston transport exports - so spans add no new exposure. userId and the rest stay off. Do not promote this to a span-metrics dimension: tenant ids are exactly the cardinality that wrecks Prometheus. Tempo indexes attributes and is built for it; Prometheus is not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(observability): document filtering traces by organization Replaces "Keep identifiers out of them", which the previous commit contradicted. The rule now distinguishes a tenant id - on spans, and already a Loki label via the winston transport, so no new exposure - from userId and emails, which stay off for the reason pg keeps enhancedDatabaseReporting: false. Records why the attribute is set twice: the root span is what Tempo's trace list and Drilldown filters actually read, and the use-case span carries the validated value. Plus the warning not to promote it to a span-metrics dimension. Also notes the incidental finding: every PackmindLogger metadata key becomes a Loki label, so per-request values like connectionId are unbounded label cardinality on the Loki side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(observability): tag traces with the space id too Same two places as the organization, since a tenant with many spaces still leaves you scanning URLs by eye. On the use-case side this needed a seam rather than a second setAttribute call: spaceId lives on SpaceMemberCommand, not PackmindCommand, so only the space-scoped bases have one. AbstractMemberUseCase now exposes a protected spanAttributes(command), which AbstractSpaceMemberUseCase and AbstractSpaceAdminUseCase override to add their own. Span concerns stay inside execute() instead of subclasses reaching for trace.getActiveSpan() from a nested call. On the root-span side the two path regexes are now compiled once at module scope - startIncomingSpanHook runs on every incoming request, so building them per call was waste. `/spaces/<uuid>` is unambiguous here: every route under organizations/:orgId/spaces takes a space id there, and spaces-management cannot match thanks to the trailing slash. Verified end to end: `{ span.packmind.space.id = "..." }` returns the trace, with both attributes on both the root HTTP span and the use-case span, and /auth/me correctly gets neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(observability): cover the space id attribute Documents `{ span.packmind.space.id = "..." }` alongside the organization filter, and why the use-case side needed a protected spanAttributes() seam: spaceId is on SpaceMemberCommand, not PackmindCommand, so only the space-scoped bases can supply one. Written up as the extension point for the next dimension worth filtering on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(observability): omit the space attribute when there is no space spaceId is optional on some commands, so the override could produce 'packmind.space.id': undefined. That was already harmless: the SDK drops undefined attribute values and the key never reaches the exporter - checked against sdk-trace-base 2.10.0, and the resulting span carries only the organization. But relying on it means a reader has to know OTel internals to be sure an absent space is fine, so the key is now spread conditionally. An absent space produces no attribute rather than an empty one, which is what keeps `{ span.packmind.space.id = "..." }` honest. Covers both space bases, with tests either way. The protected hook is reached through a passthrough on the existing test subclasses rather than standing up a tracer provider, and the absent-space case needs a cast until the command types actually make spaceId optional. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make OTEL work * Fix linting issues * ✨ feat(observability): add instrumentMethods for automatic method spans withSpan() is opt-in per call site, so anything nobody remembered to wrap is absent from the trace. instrumentMethods() walks an instance's prototype chain instead and wraps every async method, so a class opts in once from its constructor and gets depth for free - a this.b() call inside this.a() resolves through the same patched prototype. Only native AsyncFunctions are wrapped. The span has to be active while the original runs or its children become roots, so the decision cannot wait until the return value is in hand; it is made at patch time. Every build path targets es2022, so async is never downlevelled and the check holds. PACKMIND_OTEL_INSTRUMENT_METHODS=false turns it off without also losing tracing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ✨ feat(observability): span every method of every authenticated use case One instrumentMethods() call in the AbstractMemberUseCase constructor covers all 146 use cases through the three bases that extend it. validateMemberAccess and the space-membership check in executeForMembers were both invisible until now, and both do I/O. execute is skipped: it already owns an explicit span named after the class, and patching it too would nest an identical <Subclass>.execute above it. Use cases implementing IPublicUseCase directly still bypass the base class and get no span, as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ✨ feat(observability): span repository methods Names the repository method behind each pg span. Until now the driver-level spans hung off the use case as loose siblings with nothing saying which query came from where, which is exactly what makes an N+1 hard to attribute. AbstractRepository instruments itself, covering its 26 subclasses wherever they are constructed. The seven repositories that do not extend it - GitCommit, Target, Distribution, UserSpaceMembership, UserOrganizationMembership and the two provider clients - are picked up by their aggregator. The aggregators pass an explicit list rather than reflecting over their fields: they also hold a TypeORM DataSource, which must not be patched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ✨ feat(observability): span service methods Services hold the domain logic that is not a query, and they were the widest blank in the trace: a use case would show its span, then jump straight to pg with nothing in between naming the work. They share no base class, so the *Services aggregator is the seam - one instrumentComponents() call per domain. CodingAgentServices is the exception: it is a service in its own right rather than a pure aggregator, so it also instruments itself. instrumentComponents now skips nullish entries, so an optional collaborator such as EnhancedAccountsServices' apiKeyService can be listed unconditionally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 📝 docs(observability): document the automatic method-span layer The README's central claim - that packages/* is invisible unless somebody remembers to call withSpan() - stopped being true. Rewrites "Adding your own spans" around the automatic layer instead: the three call sites that apply it, why prototype patching gives depth and reaches private methods, and the three things it deliberately does not cover (sync methods, execute(), adapters and IPublicUseCase). Keeps the old 17-span capture as the "before", since it is what shows why the layer was worth adding, and adds the shape the same request now produces. That listing is derived from the code rather than pasted out of Tempo, and says so. Also carries the span-volume warning: ~17 spans becomes 60-100, which the default BatchSpanProcessor queue will silently start dropping under load, and span_name cardinality on traces_spanmetrics_* grows with it. The demo hacks stay in place for the team demo. Only the manual withSpan inside thisMethodTakesTwoSeconds goes, since the patcher now covers it - which makes that method a live proof that private methods are captured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 🐛 fix(observability): flush telemetry on fatal exits uncaughtException and unhandledRejection called process.exit(1) directly, dropping the trace and correlated logs of the fatal error itself — the ones most worth keeping. shutdownOtel never rejects and is bounded by a 2s race, so awaiting it cannot stall the exit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ✨ feat(observability): name a method span after its class alone instrumentMethods names every span `Class.method`, but a use case's entry point is that class's whole story - which is why AbstractMemberUseCase skips `execute` and wraps it by hand as a bare `<Subclass>` span instead. The `bare` option makes that shape reachable without a base class, so the use cases that extend nothing can produce spans one TraceQL query matches alongside the authenticated ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ✨ feat(observability): add instrumentUseCase and instrumentUseCases Roughly a third of the monorepo's use cases extend no base class - they implement IUseCase or nothing - so no constructor opts them in and they emit no span at all. instrumentComponents cannot close that: an adapter holds up to forty use cases in forty fields, and a list that long drifts the first time somebody adds one, which is how the gap opened. instrumentUseCases reflects instead, selecting on the value's class name rather than the field name because GitAdapter calls its fields _addGitProvider and _commitToGit. Use cases that do extend a base class cost nothing: their prototype is already marked, so the sweep is a no-op. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 🗑️ remove(observability): stop tracing Postgres at the driver level Datadog Database Monitoring takes over query-level observability. It reads pg_stat_statements and expects dd-trace, so it cannot consume OTel spans anyway - and meanwhile one repository call expanded into eight-plus pg.query and pg-pool.connect children, burying the first-party spans that say what the request was actually doing. There was nothing to delete: the pg spans came from getNodeAutoInstrumentations defaults, so this is a disable flag. One flag covers every row, because the package registers a pg AND a pg-pool module definition. It also stops the pg connection-pool metrics, which DBM reports instead. Slow queries stay visible one level up: instrumentMethods already spans every repository method, so a slow SELECT surfaces as a slow <Name>Repository.<method> span, without the statement text. Comments citing pg spans or enhancedDatabaseReporting are corrected in the same pass. The load-order warnings stay - still load-bearing for winston, express and ioredis. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 🗑️ remove(observability): drop the database latency dashboard panel "Database call latency" charted SPAN_KIND_CLIENT and described itself as "pg.query and friends". With the pg instrumentation off it would render a near-empty panel making a claim it can no longer support. The four remaining panels are all SPAN_KIND_SERVER and unaffected, as is the slow-request alert rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ⚡️ perf(skills): revert the demo slow-query scaffolding The span demo put a real (SELECT pg_sleep(2)) IS NOT NULL predicate in the skills SELECT and a two-second sleep in the use case, both marked DEMO ONLY. Together they cost every space skills listing about four seconds. The pg_sleep stub in makeTestDatasource goes with it - it existed only because pg-mem has no native pg_sleep, so the demo query would otherwise fail outright in repository specs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 📝 docs(observability): drop the two-second demo from the span guide The "What it looks like" captures were built around thisMethodTakesTwoSeconds and the pg_sleep predicate, both now reverted, so the listings referenced a method that no longer exists and showed a two-second request that no longer happens. - Remove the demo span from both waterfalls, and subtract the 2001ms it contributed to every ancestor in the "before" capture; leaf timings are untouched and the subtraction is stated in the text - Correct the span counts the demo inflated: 17 spans becomes 16, and the pg siblings were eleven, not thirteen - Repoint the private-method proof at fetchUser, which is genuinely private AND inherited from AbstractMemberUseCase, so it makes the same point without demo scaffolding - Fix the span-volume advice, which told the reader to drop the repository layer because pg spans cover that ground - with pg disabled those spans are now the only record of database work in a trace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ✨ feat(observability): span use cases that have no base class 68 of the monorepo's use cases extended nothing and so emitted no span at all - the whole sign-in surface in accounts, most of deployments and git. A trace through one of them dropped straight from the Nest handler to the repository layer, which is the blank the method-span layer exists to fill. The adapter is the seam: nine of them build the domain's use cases at the end of initialize(), so one sweep each covers them. SpacesAdapter builds one per call instead and wraps at the `new` site, as does the one job factory that does its own wiring. Nothing fails at runtime when an adapter forgets the call - the traces just stop one level short, silently, which is how this gap opened. So the rule is now part of the IBaseAdapter contract and checked by a test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 📝 docs(observability): document the adapter use-case sweep The README said use cases without a base class "get no span at all, as before", which is now the opposite of true. Records the adapter as the fourth call site, why it reflects where instrumentComponents does not, and the three things that stayed uneven: the domain-named entry points that report qualified, the non-async adapter methods, and the tenant attribute only AbstractMemberUseCase sets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 📝 docs(observability): fix the pg claims and cut the guide down The guide still described a stack that traced Postgres at the driver level. It promised db.query.text on pg.query spans, explained bind-value redaction and enhancedDatabaseReporting, showed waterfalls made mostly of pg rows, offered TraceQL recipes matching name =~ "pg.query.*", and drew a service graph with a postgres node. None of that exists any more. Fixes: - Say up front that Postgres is not traced and why, and that database work now appears as the repository-method span that issued it - Rewrite the waterfall and cookbook sections around that: one code-derived trace shape instead of three pg-heavy captures, SkillRepository.* as the slow-database-work query, redis and the LLM APIs as the only client spans - Drop the db.query.text and bind-value sections outright, and the instrumented table's Postgres row, replaced by an explicit gap entry - Mark the Grafana Cloud volume measurement as out of date: it predates both the method-span layer and this change, so only the shape of the finding still holds - Attribute the 24-spans-to-6 figure to the middleware and router reductions together, not to middleware alone Simplification: 693 lines to 495. Merged the three backend sections into one, folded the five numbered Grafana walkthroughs into a waterfall guide plus a single query cookbook, and compressed the multi-environment and alerting prose. Every measured or verified finding is kept, just shorter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 📝 docs(observability): document the Datadog OTLP target - Record Datadog EU1 direct OTLP intake, with the env vars it needs - State why dd-trace is rejected in favour of the vendor-neutral SDK - Note the per-org 403 gate and the Sentry tracer-provider collision - Keep the local Grafana stack as a documented fallback Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 🔥 remove(observability): drop the Datadog DBM scaffolding and env backups The postgres `command:` override (pg_stat_statements, compute_query_id, track_activity_query_size) and its docker/postgres/initdb mount belonged to the database-latency work already reverted in 96d4526 and bc9b9a7. The mount pointed at an untracked directory, so it created a stray directory on any other machine. Also drop scripts-dd-smoke.sh, a one-off that greped the API key out of .env to confirm the EU1 org approval. That finding is recorded in the README now. Widen the .env rule to .env.* so an ad-hoc backup can never be committed: .env matched exactly, and a .env.bak-* holding a real key was one `git add -A` away from CI, where GitGuardian runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ✨ feat(observability): ship the alert rule the compose file already mounts docker-compose.yml has been bind-mounting docker/otel/grafana/provisioning-alerting.yaml since the otel-lgtm service landed, but the file was never committed. On any other machine Docker created a directory at that path instead, so Grafana provisioned no contact point and no rule: the alerting the README documents did not exist outside one working copy. The rule queries Prometheus rather than Tempo because Tempo's plugin declares `alerting: false`, and its threshold is in seconds because traces_spanmetrics_latency_* is a seconds histogram. Also drop "the browser posts here directly" from the 4318 port comment. Browser tracing was reverted in 7e72083; the comment outlived it and contradicted the README, which states the browser is deliberately not instrumented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * 🐛 fix(observability): stop Sentry from displacing the tracer provider @sentry/nestjs is itself OpenTelemetry-based and registers a provider of its own. The two have never met, because each is gated on a different variable: SENTRY_DSN_API is unset locally, OTEL_EXPORTER_OTLP_ENDPOINT is unset in production. Turning on OTLP export in production would have been the first time both ran in one process, and Sentry — whose init resolves its DSN asynchronously, so it always runs last — would have displaced the provider otel.ts installed. Silently: the API keeps serving and the traces just stop. otel.ts now exports `otelStarted`, and instrument.ts forwards it as Sentry's `skipOpenTelemetrySetup`, so OpenTelemetry owns tracing whenever it is exporting and Sentry narrows to error reporting. The three combinations that work today are untouched; only "both set" changes. Sentry's own OTel helpers are deliberately left out. SentrySampler defers to the client's `tracesSampleRate`, which is unset — and an unset rate means tracing is disabled, which measurably drops every span, including the ones bound for OTLP. That would have reintroduced the same silent failure from the other side. SentryPropagator and SentryContextManager go with it, since an OTLP backend expects the default W3C propagator. So no @sentry/opentelemetry dependency and nothing to keep version-matched. The cost, documented in the README: Sentry issues carry no trace id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ♻️ refactor(observability): fold the space span attribute into the base AbstractSpaceMemberUseCase and AbstractSpaceAdminUseCase both extend AbstractMemberUseCase and neither extends the other, so the spaceId span attribute lived as two verbatim overrides, comments included — the shape that drifts the moment one is edited. AbstractMemberUseCase now reads spaceId off the command itself, and both overrides are gone. The existing specs assert the attributes and are unchanged, which is what shows the behaviour is identical. Also widen instrumentUseCases.arch.spec.ts to apps/api/src. It builds no use cases today, so the rule is not currently violated there — the point is that it cannot start to without the guard noticing. Declare @opentelemetry/sdk-node as a devDependency: two specs import it and it was resolving only through the workspace-root hoist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 49d9c2b commit 135d63c

57 files changed

Lines changed: 4342 additions & 72 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,10 @@ DECISIONS.md
4848
LEARNINGS.md
4949
test-results/
5050
.env
51+
# Any .env variant too: ad-hoc backups (.env.bak-*, .env.local) hold real
52+
# credentials, and GitGuardian runs on CI. Negate explicitly if a committed
53+
# template is ever added.
54+
.env.*
5155
compose.env
5256
.mcp.json
5357
!scripts/michel/.mcp.json

apps/api/docker-package.json

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,16 @@
1919
"@nestjs/jwt": "^11.0.2",
2020
"@nestjs/platform-express": "^11.1.17",
2121
"@nestjs/typeorm": "^11.0.0",
22+
"@opentelemetry/api": "^1.9.1",
23+
"@opentelemetry/auto-instrumentations-node": "^0.79.0",
24+
"@opentelemetry/exporter-logs-otlp-http": "^0.221.0",
25+
"@opentelemetry/exporter-trace-otlp-http": "^0.221.0",
26+
"@opentelemetry/instrumentation-express": "^0.69.0",
27+
"@opentelemetry/resources": "^2.10.0",
28+
"@opentelemetry/sdk-logs": "^0.221.0",
29+
"@opentelemetry/sdk-node": "^0.221.0",
30+
"@opentelemetry/semantic-conventions": "^1.43.0",
31+
"@opentelemetry/winston-transport": "^0.31.0",
2232
"@sentry/nestjs": "^9.40.0",
2333
"@sentry/node": "^9.40.0",
2434
"@teppeis/multimaps": "^3.0.0",

apps/api/src/instrument.ts

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,39 @@
1+
// MUST stay first: OpenTelemetry patches modules as they are required, so it
2+
// has to run before anything below pulls in winston, express or ioredis.
3+
import { otelStarted } from './otel';
4+
15
import { Configuration } from '@packmind/node-utils';
26
import { PackmindLogger } from '@packmind/logger';
37
import * as Sentry from '@sentry/nestjs';
48

59
Configuration.getConfig('SENTRY_DSN_API').then((sentryDSN) => {
610
if (sentryDSN) {
7-
new PackmindLogger('Sentry').info('Initializing Sentry');
11+
new PackmindLogger('Sentry').info('Initializing Sentry', {
12+
skipOpenTelemetrySetup: otelStarted,
13+
});
814
Sentry.init({
915
dsn: sentryDSN,
1016
environment: process.env.NODE_ENV || 'development',
17+
// Sentry is OpenTelemetry-based and would register a tracer provider of
18+
// its own, displacing the one ./otel already installed — leaving the API
19+
// serving traffic with no traces and nothing in the logs to say so. So
20+
// when OTLP export is on, OpenTelemetry owns tracing and Sentry is
21+
// narrowed to error reporting.
22+
//
23+
// Deliberately NOT paired with Sentry's SentrySampler: it defers to the
24+
// client's `tracesSampleRate`, which is unset here, and an unset rate
25+
// means "tracing disabled" — measured, that drops every span, including
26+
// the ones bound for OTLP. SentryPropagator and SentryContextManager are
27+
// left out for the same reason, that tracing is not Sentry's job here:
28+
// the default W3C propagator is what an OTLP backend expects.
29+
//
30+
// The cost is that Sentry issues carry no trace id, so there is no
31+
// click-through between a Sentry issue and its trace.
32+
//
33+
// False when OTLP is off, which is production today: Sentry then sets up
34+
// its own OpenTelemetry exactly as before, and nothing about this path
35+
// changes.
36+
skipOpenTelemetrySetup: otelStarted,
1137
});
1238
} else {
1339
new PackmindLogger('Sentry').info('Sentry not initialized');

apps/api/src/main.ts

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ import { NestFactory } from '@nestjs/core';
1919
import bodyParser from 'body-parser';
2020
import cookieParser from 'cookie-parser';
2121
import { AppModule } from './app/app.module';
22+
import { shutdownOtel } from './otel';
2223
import { PackmindLogger, LogLevel } from '@packmind/logger';
2324
import { Configuration, Cache } from '@packmind/node-utils';
2425
import { enableAmplitudeProxy } from '@packmind/editions';
@@ -185,16 +186,22 @@ async function bootstrap() {
185186
shutdownTime: new Date().toISOString(),
186187
totalUptime: process.uptime(),
187188
});
189+
190+
// Last, because it must be the final thing before process.exit:
191+
// exiting with an export in flight drops the whole buffered batch.
192+
await shutdownOtel();
188193
process.exit(0);
189194
})
190-
.catch((error) => {
195+
.catch(async (error) => {
191196
const errorMessage =
192197
error instanceof Error ? error.message : String(error);
193198
logger.error('❌ Error during graceful shutdown', {
194199
signal,
195200
error: errorMessage,
196201
shutdownTime: new Date().toISOString(),
197202
});
203+
204+
await shutdownOtel();
198205
process.exit(1);
199206
});
200207
};
@@ -211,7 +218,10 @@ async function bootstrap() {
211218
stack: error.stack,
212219
pid: process.pid,
213220
});
214-
process.exit(1);
221+
// Flush first: the trace and logs of a fatal error are the ones worth
222+
// keeping. shutdownOtel never rejects and is time-bounded, so this
223+
// cannot stall the exit.
224+
void shutdownOtel().finally(() => process.exit(1));
215225
});
216226

217227
process.on('unhandledRejection', (reason) => {
@@ -221,7 +231,7 @@ async function bootstrap() {
221231
reason: errorMessage,
222232
pid: process.pid,
223233
});
224-
process.exit(1);
234+
void shutdownOtel().finally(() => process.exit(1));
225235
});
226236

227237
Logger.log(

apps/api/src/otel.spec.ts

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
/**
2+
* The gate matrix for `otelStarted`, which `instrument.ts` forwards as Sentry's
3+
* `skipOpenTelemetrySetup`. Nothing about getting this wrong is loud: two tracer
4+
* providers leave the API serving traffic with no traces and no error, which is
5+
* exactly the failure this flag exists to prevent.
6+
*
7+
* `jest.isolateModulesAsync` because otel.ts decides everything at module load
8+
* from `process.env`, so each case needs a fresh evaluation.
9+
*/
10+
describe('otelStarted', () => {
11+
const originalEnv = process.env;
12+
13+
beforeEach(() => {
14+
process.env = { ...originalEnv };
15+
delete process.env['OTEL_EXPORTER_OTLP_ENDPOINT'];
16+
delete process.env['OTEL_RESOURCE_ATTRIBUTES'];
17+
});
18+
19+
afterEach(() => {
20+
process.env = originalEnv;
21+
jest.clearAllMocks();
22+
});
23+
24+
const loadOtel = async (): Promise<boolean> => {
25+
let started = false;
26+
await jest.isolateModulesAsync(async () => {
27+
const otel = await import('./otel');
28+
started = otel.otelStarted;
29+
await otel.shutdownOtel();
30+
});
31+
return started;
32+
};
33+
34+
describe('when no endpoint is configured', () => {
35+
it('does not start the SDK', async () => {
36+
await expect(loadOtel()).resolves.toBe(false);
37+
});
38+
});
39+
40+
describe('when an endpoint is configured with an environment', () => {
41+
it('starts the SDK', async () => {
42+
process.env['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'http://localhost:4318';
43+
process.env['OTEL_RESOURCE_ATTRIBUTES'] =
44+
'deployment.environment.name=test';
45+
46+
await expect(loadOtel()).resolves.toBe(true);
47+
});
48+
});
49+
50+
describe('when an endpoint is configured without an environment', () => {
51+
it('refuses to start rather than mislabel the deployment', async () => {
52+
jest.spyOn(console, 'error').mockImplementation(() => undefined);
53+
process.env['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'http://localhost:4318';
54+
55+
await expect(loadOtel()).resolves.toBe(false);
56+
});
57+
});
58+
59+
describe('when the environment is declared among other resource attributes', () => {
60+
it('starts the SDK', async () => {
61+
process.env['OTEL_EXPORTER_OTLP_ENDPOINT'] = 'http://localhost:4318';
62+
process.env['OTEL_RESOURCE_ATTRIBUTES'] =
63+
'service.version=dev,deployment.environment.name=staging';
64+
65+
await expect(loadOtel()).resolves.toBe(true);
66+
});
67+
});
68+
});

0 commit comments

Comments
 (0)