Commit 135d63c
chore(observability): add grafana/otel-lgtm to the local dev stack (#431)
* chore(observability): add grafana/otel-lgtm to the local dev stack
Adds a self-hosted OTLP backend (OTel Collector + Tempo + Prometheus + Loki +
Grafana in one container) for local trace analysis, behind an `observability`
compose profile so it does not weigh on the default dev stack.
Tracing is off unless OTEL_EXPORTER_OTLP_ENDPOINT / VITE_OTEL_EXPORTER_URL are
set, which keeps the exporter from retrying against a host that is not running.
No custom collector config is mounted: the image already enables CORS on the
OTLP/HTTP receiver, which is all the browser exporter needs. The image tag is
pinned because its Grafana provisioning and collector config are internal
details that change between releases.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* feat(observability): trace the API with OpenTelemetry down to pg queries
Adds apps/api/src/otel.ts, starting the OTel NodeSDK with the auto-instrumentation
bundle and exporting OTLP. Gives request -> Nest controller -> SQL statement
spans, plus Redis and outgoing LLM calls (the OpenAI/Anthropic/Google SDKs use
global fetch, which the http instrumentation does not patch - undici does).
The SDK starts synchronously at module top level and otel.ts is the first import
of instrument.ts. Both matter: auto-instrumentation patches modules as they are
required, so anything loaded first is never traced. That is also why otel.ts
imports no @packmind/* module - @packmind/logger pulls in winston, and a winston
required ahead of the hooks would lose log/trace correlation. Verified on the
built bundle: the [otel] line prints before any application log, and Winston
records then carry trace_id/span_id.
Everything is gated on OTEL_EXPORTER_OTLP_ENDPOINT, which is unset in production
and in tests, so the SDK never starts there.
pg keeps enhancedDatabaseReporting off, so spans carry parameterized SQL but
never bind values, and GenAI message content is left uncaptured - neither user
data nor prompts reach the trace backend.
OTel packages are added to apps/api/docker-package.json as well, since that file
is hand-maintained and a missing entry breaks the production container.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* feat(logger): surface trace ids in console log output
@opentelemetry/instrumentation-winston injects trace_id, span_id and trace_flags
into every record logged inside an active span, so correlation needs no code
here. But those three fields landed in the JSON metadata blob of every single
console line, drowning the actual payload.
Renders a short [trace=xxxxxxxx] marker instead and keeps the three fields out
of the blob. The json() format still carries them in full, which is what Loki
and Grafana's trace<->logs navigation use.
Extracts the printf callback into an exported formatConsoleLine so it can be
tested directly: the console transport is unreachable under Jest, which sets
PACKMIND_LOG_LEVEL=silent globally.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* feat(frontend): emit browser traces over OTLP
Adds OtelService with a WebTracerProvider exporting OTLP, so a page interaction
and the API work it triggers land on a single trace. Both XHR and fetch are
instrumented: Axios uses the XHR adapter in the browser, so XHR is what covers
packmindApiService, while fetch covers React Router's own requests.
Initialization lives in entry.client.tsx rather than root.tsx, where initSentry
sits: React Router runs in SPA mode but still prerenders the shell at build
time, so root.tsx module scope executes in Node, where window and
XMLHttpRequest do not exist. Confirmed on a real build - the prerender logs
Sentry's message but not this one, and the OTel code lands only in the
entry.client chunk.
No CORS or propagateTraceHeaderCorsUrls config is needed: the API is reached
through the relative /api path, so requests are same-origin and traceparent is
attached by default. Only the exporter POST is cross-origin, which the collector
already allows.
Gated on VITE_OTEL_EXPORTER_URL, unset by default.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* fix(observability): flush OpenTelemetry before the process exits
Review feedback on #431: the SIGTERM/SIGINT listeners in otel.ts started
sdk.shutdown() without awaiting it, while main.ts's own listener ended its
sequence with process.exit - which terminates regardless of an in-flight export.
Reproduced against the built bundle with a stub OTLP collector, real postgres
and redis: sending SIGTERM inside the BatchSpanProcessor window produced zero
exports, and a control run confirmed spans were being generated (a 22KB batch
exported once the 5s window elapsed). So it was not a partial race - the entire
final batch was lost, every time.
Replaces the listeners with an exported shutdownOtel() that main.ts awaits as
the last step before process.exit, in both the success and failure paths. Two
listeners racing each other cannot be made correct; shutdown ordering belongs
where it already lives. The flush is bounded by a 2s timeout so an unreachable
collector cannot hold the process past a container's SIGKILL grace, and it never
rethrows, so a failed flush cannot change the exit path.
Verified after the change: trace export lands 26ms after SIGTERM and the process
exits at 205ms; with the collector unreachable, shutdown completes in 2.2s
instead of hanging on exporter retries.
Also documents the Node runtime metrics the SDK exports by default, which the
collector stub revealed and the README did not mention.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* chore(observability): bind otel-lgtm ports to loopback
Review feedback on #431. OTLP ingestion is unauthenticated and Grafana ships on
default credentials, while traces carry SQL statements, request URLs and log
lines - so publishing on every interface exposed all of it to the local network.
Functionally free: the browser reaches the collector over loopback, and the
backend reaches it as http://otel-lgtm:4318 over the compose network, which host
port bindings do not affect. Confirmed via `docker compose config` that all
three now resolve to host_ip 127.0.0.1.
Documents the choice in both the compose file and the README, since the ports
will look dead to anyone driving Docker from another machine.
Note this does not make the dev stack safe on a hostile network: postgres,
redis and pgadmin are still published on 0.0.0.0 with weak or absent
credentials. Hardening those is a wider change than this PR.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* ci(observability): inject browser tracing endpoint for Cloud builds only
VITE_OTEL_EXPORTER_URL is baked into the client bundle by Vite and cannot be
changed after the image is built, so the Cloud/self-hosted split has to happen at
build time. Gates it on the same expression the Sentry and Crisp values already
use - proprietary edition AND not a release/* tag - since every release/* tag
produces the self-hosted images.
Adds a guard that fails the frontend build if any Cloud-only VITE_ value is
non-empty on a release/* tag. The failure it prevents is invisible: a leaked
endpoint in a customer's bundle would silently point their browsers at Packmind
infrastructure, and nothing in the running product would look wrong. It covers
the Sentry and Crisp values too, which had no such check.
Verified both directions by building the frontend with a sentinel URL: the value
reaches the bundle when set, and with the empty value the gate produces the
bundle contains VITE_OTEL_EXPORTER_URL:"" and initOtel() returns early. Worth
checking because getEnvVar reads import.meta.env with a dynamic key, which Vite
can only satisfy by serializing the whole env object.
The API needs no equivalent change - OTEL_EXPORTER_OTLP_ENDPOINT is read at
runtime, so the image is neutral and self-hosted deployments never start the SDK.
Documents that in the self-hosted compose file, and the whole split in
docker/otel/README.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* docs(observability): correct the pg span attribute name to db.query.text
Captured real spans against a migrated postgres to check what the pg
instrumentation actually records, and the SQL is not under `db.statement` as
documented - instrumentation-pg has moved to the stable database semantic
conventions, so it is `db.query.text`, alongside `db.system.name` and
`db.namespace`. Anyone following the old name would have concluded the SQL was
not captured at all.
The substance was right: the full TypeORM-generated statement is recorded, and
bind values are not. Confirmed by calling check-email-availability with a real
address and finding only `LOWER("user"."email") = LOWER($1)` in the span, with
the address nowhere in it. Documents that example, since "parameterized" is
easier to trust when you can see what it looks like.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* fix(observability): cut a request from 24 spans to 6
Captured a real trace for one endpoint call and it was close to unreadable: 24
spans, of which 18 were Express middleware and routing noise, mostly named
"middleware - patched", burying the four spans that carry the story.
Two causes:
- Per-middleware spans (cookieParser, jsonParser, cors...) each wrapped in an
uninformative parent. Dropped via ignoreLayersType.
- Express 5 routes through the standalone `router` package, which the auto
bundle instruments separately - so routing was traced twice, contributing 9
opaque spans plus a duplicate of the request-handler span express already
emits. Disabled; express covers the same ground.
The same request now produces 6 spans reading HTTP -> route -> controller -> use
case -> SQL, which is the shape someone new to tracing can actually follow.
Declares @opentelemetry/instrumentation-express directly rather than relying on
it transitively, since pnpm's isolated node_modules would not resolve it at
runtime, and adds it to docker-package.json for the production image.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* fix(observability): actually export logs, so trace correlation works
Verified the correlation end to end by capturing traces and logs from one
request and cross-referencing, and found that no log records were being exported
at all. The trace_id visible in console output came from the injection half of
instrumentation-winston, which made it look like everything worked.
Cause: log sending needs @opentelemetry/winston-transport, which is an OPTIONAL
peer of instrumentation-winston. Without it the instrumentation sends nothing and
only emits an OTel diag warning - invisible unless OTEL_LOG_LEVEL is set. So
Loki would have stayed empty, and the Tempo<->Loki click-through that justified
enabling log sending in the first place would not have existed.
Adds the package to the root dependencies and to docker-package.json, and warns
against pruning it in both the code comment and the README.
Confirmed after the fix, on a single request:
- 5 log records carry the same trace_id as the HTTP span, with span ids
belonging to spans in that trace
- startup logs correctly carry no trace context
- an inbound W3C traceparent is adopted: the server span takes the caller's
trace id and parent span id, and the request's log records carry the caller's
trace id too - which is the browser -> API -> SQL -> logs path
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* docs(observability): document TraceQL duration queries
Adds the "show me everything slower than X" entry point, and the distinction
that catches people out: duration filters a single span, traceDuration filters
the whole request, so {duration > 800ms} finds a slow query even inside a fast
request.
Verified against a real Tempo 3.0.3 and Loki 3.7.6 running locally, fed by the
built API through a splitter standing in for the collector. Looking up one trace
id returned its 5 log lines from Loki and all 6 spans from Tempo, and the
duration filters selected exactly the slow traces.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* docs(observability): document latency percentiles per operation
Answers "distribution time with percentiles for each unique method": Tempo's
metrics-generator emits traces_spanmetrics_latency_bucket labelled by span_name,
so percentiles per endpoint need no extra instrumentation - only a
histogram_quantile query.
Verified end to end rather than described, by running Tempo 3.0.3 with the
span-metrics processor remote-writing to Prometheus 3.13.2 and driving the built
API with mixed fast and lock-delayed traffic. Measured, from that run:
span p50 p95 p99
POST /../check-email-availability 3.1ms 46.4ms 216.3ms
pg.query:SELECT packmind 1.1ms 21.6ms 213.8ms
which is also the example the docs use, since the matching p99s show the tail
sitting in the query rather than in our code.
Records two gotchas found while testing: rate() with no traffic in the window
returns NaN, so a quiet environment looks broken when it is not, and unmatched
routes collapse into a single bare "POST" span name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* feat(observability): ship a ready-made Grafana dashboard
Writing PromQL and TraceQL by hand for every question is too much friction for
day-to-day use, so the queries are now written once and provisioned.
Adds docker/otel/grafana/dashboards/packmind-api.json - latency percentiles per
endpoint, request and error rates, a latency heatmap with an endpoint selector,
and database call percentiles - plus the provisioning file and the compose
mounts. The otel-lgtm image provisions datasources but ships no dashboards, so
this fills the gap; datasource uids (prometheus/tempo/loki) are stable in the
image, so the panels bind without manual selection.
Also documents that the image already installs the Drilldown apps
(exploretraces, lokiexplore, metricsdrilldown), which are point-and-click and
cover most exploration with no query language at all. The README now leads with
those and frames the query sections as the escape hatch rather than the entry
point.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* docs(observability): warn about span-metric naming fragmentation
Importing a community dashboard from grafana.com is the obvious reflex, and it
will usually show empty panels. Span-metric names differ across the ecosystem:
this stack's Tempo metrics-generator emits traces_spanmetrics_latency_bucket
(confirmed by querying Prometheus directly), while the Collector's spanmetrics
connector and Alloy emit traces_spanmetrics_duration_milliseconds_bucket, and
older Tempo used traces_spanmetrics_duration_seconds_bucket.
Documents the mismatch and the two ways out, so an empty panel does not get read
as a broken pipeline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* docs(observability): make the dashboard portable and document the Cloud move
"Just change the endpoint" covers the API and nothing else, so this writes down
what actually has to happen.
Verified: the SDK reads OTEL_EXPORTER_OTLP_HEADERS natively, so authenticating
to Grafana Cloud needs no code change. Confirmed by pointing the built API at an
endpoint requiring auth and observing Authorization: Basic on /v1/traces and
/v1/logs.
Not covered by an endpoint change, and now documented:
- browser traces cannot go direct, since VITE_OTEL_EXPORTER_URL is resolved by
the browser and would expose the Cloud token in a public bundle
- span metrics are off by default in Cloud and billed as active series, so the
RED panels stay empty until enabled
- nothing is sampled and every log line is exported, which is fine on a laptop
and expensive in production
Also replaces the dashboard's hardcoded datasource uid with a ${ds} variable, so
the same JSON works against otel-lgtm and against Cloud, where uids differ.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* revert(observability): drop browser tracing, keep it to the API
Removes the frontend side of the OpenTelemetry work: OtelService, its call in
entry.client.tsx, the four browser-only packages, VITE_OTEL_EXPORTER_URL in
compose and in both workflows, and the matching documentation.
Sending telemetry from a page has no good answer here. The endpoint has to be
reachable by the browser, so it either ships a credential in a public bundle or
becomes an unauthenticated ingest path on our own domain - and fetching the
credential from an API changes nothing, since anything the page can read, anyone
can. Neither trade was worth what browser spans add on top of server-side
tracing.
Keeping it server-side also removes a whole class of problem: the API reads its
endpoint at runtime, so the image is neutral and no build-time gate is needed to
keep telemetry out of self-hosted images. The CI guard stays for the Sentry and
Crisp values, which are still baked into the bundle and had no such check
before.
Inbound traceparent handling is untouched, so an instrumented caller can still
join the trace if browser tracing is ever revisited.
Verified: frontend typecheck, lint, 1070 tests and a from-scratch build all
pass, and the rebuilt client bundle contains no OpenTelemetry code. API
typecheck and build unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* feat(observability): require an explicit environment before exporting
deployment.environment.name used to fall back to NODE_ENV, which the API image
hardcodes to production. Staging would therefore have announced itself as
production, merging its traces, logs and latency percentiles into the production
ones with nothing looking wrong.
The environment now has to be declared in OTEL_RESOURCE_ATTRIBUTES, which is
where the SDK's own resource detector reads it, so there is a single source of
truth and no default that can silently win. With an endpoint set and no
environment, the SDK does not start and logs an error.
The API still boots and serves traffic in that case: a telemetry misconfiguration
must not take the service down, while mislabelled telemetry is worse than none -
hence loud, but never fatal.
Verified on the built bundle:
- endpoint set, environment missing -> API up, error logged, collector receives
nothing
- endpoint + environment=staging -> exports with deployment.environment.name=staging
- no OTEL_* variables at all -> API up, no output, no spurious error
Compose defaults the local value so local remains a one-variable opt-in, and the
README gains the per-environment table. It also records that the attribute does
not reach span metrics, so two environments sharing one Prometheus would blend
their percentiles.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* docs(observability): correct the multi-environment guidance
The previous wording said the environment attribute cannot reach span metrics
and implied separate stacks were the only reliable answer. Grafana Cloud's
metrics-generator can promote span and resource attributes to dimensions, so
deployment.environment.name can become a label on traces_spanmetrics_* on a
single stack - with the caveat that each dimension multiplies active series and
is billed.
Documents both options and when each is the right trade, plus the multi-stack
datasource that queries several stacks at once. Notes that the bundled
dashboard's ${ds} variable is what makes one JSON serve both environments.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* docs(observability): rule out fetching OTel config at runtime, fix sampling advice
Two corrections to guidance, both in areas someone would reasonably get wrong.
Configuration.getConfig() must not be used for the OTel bootstrap. It is async
and may call Infisical, so the SDK would start only once that promise resolved -
after pg, ioredis, express and winston were required, and therefore never
patched. Nothing crashes, there are simply no spans. The Sentry init in
instrument.ts already demonstrates the trap, its DSN resolving after bootstrap
has begun. Records that these have to be real environment variables, and that
the only sensitive one can come through the entrypoint mechanism the container
already uses for its other secrets.
Replaces the "add traceidratio 0.1" advice, which was too blunt: head sampling
drops 90% of errors and slow requests along with everything else, and Tempo's
metrics-generator only sees what arrives, so percentiles end up computed on the
sample. Documents tail sampling in a collector instead - 100% in, decide once
the trace is complete, metrics computed before the sampling stage.
Backs it with a measurement: 50 real requests on the simplest endpoint cost
6.4 KB of traces and 7.4 KB of logs each, uncompressed. Logs are the bigger
half, and trace sampling does nothing about them - so filtering logs buys more
than sampling does, which reverses the priority previously given.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* revert(ci): restore build.yml, its guard no longer belongs here
The frontend build guard was added to keep the browser OTLP endpoint out of
self-hosted bundles. That variable is gone, so what remained only protected the
Sentry and Crisp values - which were already gated by the same expressions
before this branch existed, and which this PR was never about.
build.yml is now byte-identical to main, and main.yml already was. The CI diff
of this PR is empty.
The gap it covered is real: nothing verifies that the gate expressions actually
resolved to empty on a release/* tag, and a mistake there is invisible once
bundled. That deserves its own change rather than riding along with API tracing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NJvWXKCQBSLAEps8nZS55s
* feat(observability): add withSpan, the first manual span helper
Auto-instrumentation only patches known library modules - http, express,
nestjs-core, pg, winston - so everything in packages/* is invisible in a
trace. A waterfall drops from the Nest handler span straight to pg.query
with the services, adapters and use cases in between unaccounted for.
withSpan() closes that gap. @opentelemetry/api was already a declared
dependency imported nowhere; this is its first caller.
It degrades cleanly: with no SDK registered - unit tests, or the API
started without OTEL_EXPORTER_OTLP_ENDPOINT - trace.getTracer() returns a
no-op tracer, the callback still runs, and the cost is a function call.
Not usable from apps/api/src/otel.ts: the bootstrap must not import
@packmind/*, or winston loads before sdk.start() and log/trace
correlation breaks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(observability): give every use case a span
Between the Nest handler span and the pg spans the waterfall was blank -
services, hexa adapters and use cases all invisible - so a slow request
told you it was slow without telling you where.
AbstractMemberUseCase.execute() is the single chokepoint for every
authenticated use case: AbstractAdminUseCase, AbstractSpaceMemberUseCase
and AbstractSpaceAdminUseCase all extend it and inherit this method. One
withSpan() here names the whole domain layer, forever, with no per-use-case
opt-in.
The span carries no attributes on purpose. userId and organizationId would
put identifiers in the trace backend, the same reason pg keeps
enhancedDatabaseReporting: false. The class name is the payload.
Costs roughly one span per request against the six documented in
docker/otel/README.md.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(observability): add a demo slow span to the space skills list
DEMO ONLY - revert this commit once the Grafana screenshots are taken.
Gives the OTLP setup something unmistakable to look at: a span named
thisMethodTakesTwoSeconds, on GET /organizations/:orgId/spaces/:spaceId/skills,
found with `{ name = "thisMethodTakesTwoSeconds" }` in TraceQL and nested
under the ListSkillsBySpaceUseCase span from the previous commit.
The wait is awaited, not a busy-wait: the event loop stays free, so
concurrent requests are unaffected while the span still reads exactly like
a genuinely slow call.
Do not let this reach a deployed environment - it is an unconditional two
second regression on the skills list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(observability): document manual spans, fix a mislabelled one
The waterfall in section 1 labelled the second Nest row "use case". It is
not: instrumentation-nestjs-core emits a REQUEST_CONTEXT span named
Controller.method and a REQUEST_HANDLER span named after handler.name
alone, so both rows are the controller. It patches
RouterExplorer.createHandler and nothing else, which means a bare method
name in a waterfall is still the route handler - and nothing from
packages/* was in that trace at all.
That mislabel is worth correcting because it invites exactly the wrong
conclusion: that the domain layer is already traced and a slow use case
would show up on its own.
Adds a section on withSpan() and the automatic use-case span, with a real
capture of the space skills endpoint. Two things worth knowing that are
easy to get wrong: the TraceQL intrinsic for instrumentation scope is
`instrumentation:name` and `scope.name` does not parse, and spans
deliberately carry no attributes so identifiers stay out of the backend.
Also drops "Express middleware" from the instrumented table - those spans
have been deliberately off since ae2f87b.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore(observability): record @opentelemetry/api in the lockfile
Follows the dependency added to packages/node-utils for withSpan. The
eslint `deprecated` line comes along because pnpm renormalised the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(observability): make the skills SQL slow too, for the demo
DEMO ONLY - revert with the rest of the span demo.
thisMethodTakesTwoSeconds proves a first-party span shows up, but says
nothing about slow SQL. This puts the delay in the real skills SELECT
instead of a synthetic sleep statement, so the pg.query span is the actual
query with the sleep visible in db.query.text - what a genuinely slow
statement looks like. Bind values stay parameterized, so nothing about the
caller leaks.
The subquery form is load-bearing. `(SELECT pg_sleep(2)) IS NOT NULL` is
uncorrelated, so Postgres hoists it to an InitPlan and runs it once -
measured at 1.0s across 5 rows. Bare as `pg_sleep(2) IS NOT NULL` it is
volatile and runs per row: 4 rows at 0.5s each took 2.0s. IS NOT NULL on
void is true, so no rows are filtered.
Needs the pg_sleep stub in test-utils because the repository specs run on
pg-mem, which implements very few native functions and fails the query
outright without one. The stub does not sleep, so the suite stays fast.
Note the blast radius: findBySpaceId also backs skill create, update and
upload duplicate-name checks and two dashboard use cases, so those pay the
two seconds as well.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(observability): tag traces with the organization id
Filtering traces per customer meant reading URLs by eye. Now
`{ span.packmind.organization.id = "..." }` does it.
Set in two places, deliberately. instrumentation-http's
startIncomingSpanHook puts it on the ROOT span, which is what Tempo's trace
list and the Drilldown filters read - an attribute buried on a child span
is findable in TraceQL but leaves the trace list unfilterable. That hook
runs at span creation, before auth, so it has only the URL to go on and
matches a strict UUID shape, keeping a literal ":orgId" or a slug out of
the attribute. AbstractMemberUseCase then sets the same attribute from the
validated command, before validation rather than after, so a rejected
request is still attributable to whoever made it.
This reverses the "no attributes" call from 568380a, on the evidence that
organizationId is already a Loki label on every log line the winston
transport exports - so spans add no new exposure. userId and the rest stay
off.
Do not promote this to a span-metrics dimension: tenant ids are exactly the
cardinality that wrecks Prometheus. Tempo indexes attributes and is built
for it; Prometheus is not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(observability): document filtering traces by organization
Replaces "Keep identifiers out of them", which the previous commit
contradicted. The rule now distinguishes a tenant id - on spans, and
already a Loki label via the winston transport, so no new exposure - from
userId and emails, which stay off for the reason pg keeps
enhancedDatabaseReporting: false.
Records why the attribute is set twice: the root span is what Tempo's trace
list and Drilldown filters actually read, and the use-case span carries the
validated value. Plus the warning not to promote it to a span-metrics
dimension.
Also notes the incidental finding: every PackmindLogger metadata key
becomes a Loki label, so per-request values like connectionId are
unbounded label cardinality on the Loki side.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(observability): tag traces with the space id too
Same two places as the organization, since a tenant with many spaces still
leaves you scanning URLs by eye.
On the use-case side this needed a seam rather than a second setAttribute
call: spaceId lives on SpaceMemberCommand, not PackmindCommand, so only the
space-scoped bases have one. AbstractMemberUseCase now exposes a protected
spanAttributes(command), which AbstractSpaceMemberUseCase and
AbstractSpaceAdminUseCase override to add their own. Span concerns stay
inside execute() instead of subclasses reaching for
trace.getActiveSpan() from a nested call.
On the root-span side the two path regexes are now compiled once at module
scope - startIncomingSpanHook runs on every incoming request, so building
them per call was waste. `/spaces/<uuid>` is unambiguous here: every route
under organizations/:orgId/spaces takes a space id there, and
spaces-management cannot match thanks to the trailing slash.
Verified end to end: `{ span.packmind.space.id = "..." }` returns the trace,
with both attributes on both the root HTTP span and the use-case span, and
/auth/me correctly gets neither.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(observability): cover the space id attribute
Documents `{ span.packmind.space.id = "..." }` alongside the organization
filter, and why the use-case side needed a protected spanAttributes() seam:
spaceId is on SpaceMemberCommand, not PackmindCommand, so only the
space-scoped bases can supply one. Written up as the extension point for
the next dimension worth filtering on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(observability): omit the space attribute when there is no space
spaceId is optional on some commands, so the override could produce
'packmind.space.id': undefined.
That was already harmless: the SDK drops undefined attribute values and the
key never reaches the exporter - checked against sdk-trace-base 2.10.0, and
the resulting span carries only the organization. But relying on it means a
reader has to know OTel internals to be sure an absent space is fine, so the
key is now spread conditionally. An absent space produces no attribute
rather than an empty one, which is what keeps
`{ span.packmind.space.id = "..." }` honest.
Covers both space bases, with tests either way. The protected hook is
reached through a passthrough on the existing test subclasses rather than
standing up a tracer provider, and the absent-space case needs a cast until
the command types actually make spaceId optional.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Make OTEL work
* Fix linting issues
* ✨ feat(observability): add instrumentMethods for automatic method spans
withSpan() is opt-in per call site, so anything nobody remembered to wrap is
absent from the trace. instrumentMethods() walks an instance's prototype chain
instead and wraps every async method, so a class opts in once from its
constructor and gets depth for free - a this.b() call inside this.a() resolves
through the same patched prototype.
Only native AsyncFunctions are wrapped. The span has to be active while the
original runs or its children become roots, so the decision cannot wait until
the return value is in hand; it is made at patch time. Every build path targets
es2022, so async is never downlevelled and the check holds.
PACKMIND_OTEL_INSTRUMENT_METHODS=false turns it off without also losing tracing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ✨ feat(observability): span every method of every authenticated use case
One instrumentMethods() call in the AbstractMemberUseCase constructor covers
all 146 use cases through the three bases that extend it. validateMemberAccess
and the space-membership check in executeForMembers were both invisible until
now, and both do I/O.
execute is skipped: it already owns an explicit span named after the class, and
patching it too would nest an identical <Subclass>.execute above it.
Use cases implementing IPublicUseCase directly still bypass the base class and
get no span, as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ✨ feat(observability): span repository methods
Names the repository method behind each pg span. Until now the driver-level
spans hung off the use case as loose siblings with nothing saying which query
came from where, which is exactly what makes an N+1 hard to attribute.
AbstractRepository instruments itself, covering its 26 subclasses wherever they
are constructed. The seven repositories that do not extend it - GitCommit,
Target, Distribution, UserSpaceMembership, UserOrganizationMembership and the
two provider clients - are picked up by their aggregator.
The aggregators pass an explicit list rather than reflecting over their fields:
they also hold a TypeORM DataSource, which must not be patched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ✨ feat(observability): span service methods
Services hold the domain logic that is not a query, and they were the widest
blank in the trace: a use case would show its span, then jump straight to pg
with nothing in between naming the work.
They share no base class, so the *Services aggregator is the seam - one
instrumentComponents() call per domain. CodingAgentServices is the exception:
it is a service in its own right rather than a pure aggregator, so it also
instruments itself.
instrumentComponents now skips nullish entries, so an optional collaborator
such as EnhancedAccountsServices' apiKeyService can be listed unconditionally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 📝 docs(observability): document the automatic method-span layer
The README's central claim - that packages/* is invisible unless somebody
remembers to call withSpan() - stopped being true. Rewrites "Adding your own
spans" around the automatic layer instead: the three call sites that apply it,
why prototype patching gives depth and reaches private methods, and the three
things it deliberately does not cover (sync methods, execute(), adapters and
IPublicUseCase).
Keeps the old 17-span capture as the "before", since it is what shows why the
layer was worth adding, and adds the shape the same request now produces. That
listing is derived from the code rather than pasted out of Tempo, and says so.
Also carries the span-volume warning: ~17 spans becomes 60-100, which the
default BatchSpanProcessor queue will silently start dropping under load, and
span_name cardinality on traces_spanmetrics_* grows with it.
The demo hacks stay in place for the team demo. Only the manual withSpan inside
thisMethodTakesTwoSeconds goes, since the patcher now covers it - which makes
that method a live proof that private methods are captured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 🐛 fix(observability): flush telemetry on fatal exits
uncaughtException and unhandledRejection called process.exit(1) directly,
dropping the trace and correlated logs of the fatal error itself — the
ones most worth keeping. shutdownOtel never rejects and is bounded by a
2s race, so awaiting it cannot stall the exit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ✨ feat(observability): name a method span after its class alone
instrumentMethods names every span `Class.method`, but a use case's entry
point is that class's whole story - which is why AbstractMemberUseCase
skips `execute` and wraps it by hand as a bare `<Subclass>` span instead.
The `bare` option makes that shape reachable without a base class, so the
use cases that extend nothing can produce spans one TraceQL query matches
alongside the authenticated ones.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ✨ feat(observability): add instrumentUseCase and instrumentUseCases
Roughly a third of the monorepo's use cases extend no base class - they
implement IUseCase or nothing - so no constructor opts them in and they
emit no span at all. instrumentComponents cannot close that: an adapter
holds up to forty use cases in forty fields, and a list that long drifts
the first time somebody adds one, which is how the gap opened.
instrumentUseCases reflects instead, selecting on the value's class name
rather than the field name because GitAdapter calls its fields
_addGitProvider and _commitToGit. Use cases that do extend a base class
cost nothing: their prototype is already marked, so the sweep is a no-op.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 🗑️ remove(observability): stop tracing Postgres at the driver level
Datadog Database Monitoring takes over query-level observability. It reads
pg_stat_statements and expects dd-trace, so it cannot consume OTel spans
anyway - and meanwhile one repository call expanded into eight-plus pg.query
and pg-pool.connect children, burying the first-party spans that say what the
request was actually doing.
There was nothing to delete: the pg spans came from getNodeAutoInstrumentations
defaults, so this is a disable flag. One flag covers every row, because the
package registers a pg AND a pg-pool module definition. It also stops the pg
connection-pool metrics, which DBM reports instead.
Slow queries stay visible one level up: instrumentMethods already spans every
repository method, so a slow SELECT surfaces as a slow
<Name>Repository.<method> span, without the statement text.
Comments citing pg spans or enhancedDatabaseReporting are corrected in the same
pass. The load-order warnings stay - still load-bearing for winston, express
and ioredis.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 🗑️ remove(observability): drop the database latency dashboard panel
"Database call latency" charted SPAN_KIND_CLIENT and described itself as
"pg.query and friends". With the pg instrumentation off it would render a
near-empty panel making a claim it can no longer support.
The four remaining panels are all SPAN_KIND_SERVER and unaffected, as is the
slow-request alert rule.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ⚡️ perf(skills): revert the demo slow-query scaffolding
The span demo put a real (SELECT pg_sleep(2)) IS NOT NULL predicate in the
skills SELECT and a two-second sleep in the use case, both marked DEMO ONLY.
Together they cost every space skills listing about four seconds.
The pg_sleep stub in makeTestDatasource goes with it - it existed only because
pg-mem has no native pg_sleep, so the demo query would otherwise fail outright
in repository specs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 📝 docs(observability): drop the two-second demo from the span guide
The "What it looks like" captures were built around thisMethodTakesTwoSeconds
and the pg_sleep predicate, both now reverted, so the listings referenced a
method that no longer exists and showed a two-second request that no longer
happens.
- Remove the demo span from both waterfalls, and subtract the 2001ms it
contributed to every ancestor in the "before" capture; leaf timings are
untouched and the subtraction is stated in the text
- Correct the span counts the demo inflated: 17 spans becomes 16, and the pg
siblings were eleven, not thirteen
- Repoint the private-method proof at fetchUser, which is genuinely private AND
inherited from AbstractMemberUseCase, so it makes the same point without demo
scaffolding
- Fix the span-volume advice, which told the reader to drop the repository layer
because pg spans cover that ground - with pg disabled those spans are now the
only record of database work in a trace
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ✨ feat(observability): span use cases that have no base class
68 of the monorepo's use cases extended nothing and so emitted no span at
all - the whole sign-in surface in accounts, most of deployments and git.
A trace through one of them dropped straight from the Nest handler to the
repository layer, which is the blank the method-span layer exists to fill.
The adapter is the seam: nine of them build the domain's use cases at the
end of initialize(), so one sweep each covers them. SpacesAdapter builds
one per call instead and wraps at the `new` site, as does the one job
factory that does its own wiring.
Nothing fails at runtime when an adapter forgets the call - the traces
just stop one level short, silently, which is how this gap opened. So the
rule is now part of the IBaseAdapter contract and checked by a test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 📝 docs(observability): document the adapter use-case sweep
The README said use cases without a base class "get no span at all, as
before", which is now the opposite of true. Records the adapter as the
fourth call site, why it reflects where instrumentComponents does not,
and the three things that stayed uneven: the domain-named entry points
that report qualified, the non-async adapter methods, and the tenant
attribute only AbstractMemberUseCase sets.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 📝 docs(observability): fix the pg claims and cut the guide down
The guide still described a stack that traced Postgres at the driver level. It
promised db.query.text on pg.query spans, explained bind-value redaction and
enhancedDatabaseReporting, showed waterfalls made mostly of pg rows, offered
TraceQL recipes matching name =~ "pg.query.*", and drew a service graph with a
postgres node. None of that exists any more.
Fixes:
- Say up front that Postgres is not traced and why, and that database work now
appears as the repository-method span that issued it
- Rewrite the waterfall and cookbook sections around that: one code-derived
trace shape instead of three pg-heavy captures, SkillRepository.* as the
slow-database-work query, redis and the LLM APIs as the only client spans
- Drop the db.query.text and bind-value sections outright, and the instrumented
table's Postgres row, replaced by an explicit gap entry
- Mark the Grafana Cloud volume measurement as out of date: it predates both the
method-span layer and this change, so only the shape of the finding still
holds
- Attribute the 24-spans-to-6 figure to the middleware and router reductions
together, not to middleware alone
Simplification: 693 lines to 495. Merged the three backend sections into one,
folded the five numbered Grafana walkthroughs into a waterfall guide plus a
single query cookbook, and compressed the multi-environment and alerting prose.
Every measured or verified finding is kept, just shorter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 📝 docs(observability): document the Datadog OTLP target
- Record Datadog EU1 direct OTLP intake, with the env vars it needs
- State why dd-trace is rejected in favour of the vendor-neutral SDK
- Note the per-org 403 gate and the Sentry tracer-provider collision
- Keep the local Grafana stack as a documented fallback
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 🔥 remove(observability): drop the Datadog DBM scaffolding and env backups
The postgres `command:` override (pg_stat_statements, compute_query_id,
track_activity_query_size) and its docker/postgres/initdb mount belonged to
the database-latency work already reverted in 96d4526 and bc9b9a7. The
mount pointed at an untracked directory, so it created a stray directory on
any other machine.
Also drop scripts-dd-smoke.sh, a one-off that greped the API key out of .env
to confirm the EU1 org approval. That finding is recorded in the README now.
Widen the .env rule to .env.* so an ad-hoc backup can never be committed:
.env matched exactly, and a .env.bak-* holding a real key was one `git add -A`
away from CI, where GitGuardian runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ✨ feat(observability): ship the alert rule the compose file already mounts
docker-compose.yml has been bind-mounting
docker/otel/grafana/provisioning-alerting.yaml since the otel-lgtm service
landed, but the file was never committed. On any other machine Docker created
a directory at that path instead, so Grafana provisioned no contact point and
no rule: the alerting the README documents did not exist outside one working
copy.
The rule queries Prometheus rather than Tempo because Tempo's plugin declares
`alerting: false`, and its threshold is in seconds because
traces_spanmetrics_latency_* is a seconds histogram.
Also drop "the browser posts here directly" from the 4318 port comment. Browser
tracing was reverted in 7e72083; the comment outlived it and contradicted the
README, which states the browser is deliberately not instrumented.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* 🐛 fix(observability): stop Sentry from displacing the tracer provider
@sentry/nestjs is itself OpenTelemetry-based and registers a provider of its
own. The two have never met, because each is gated on a different variable:
SENTRY_DSN_API is unset locally, OTEL_EXPORTER_OTLP_ENDPOINT is unset in
production. Turning on OTLP export in production would have been the first
time both ran in one process, and Sentry — whose init resolves its DSN
asynchronously, so it always runs last — would have displaced the provider
otel.ts installed. Silently: the API keeps serving and the traces just stop.
otel.ts now exports `otelStarted`, and instrument.ts forwards it as Sentry's
`skipOpenTelemetrySetup`, so OpenTelemetry owns tracing whenever it is
exporting and Sentry narrows to error reporting. The three combinations that
work today are untouched; only "both set" changes.
Sentry's own OTel helpers are deliberately left out. SentrySampler defers to
the client's `tracesSampleRate`, which is unset — and an unset rate means
tracing is disabled, which measurably drops every span, including the ones
bound for OTLP. That would have reintroduced the same silent failure from the
other side. SentryPropagator and SentryContextManager go with it, since an
OTLP backend expects the default W3C propagator. So no @sentry/opentelemetry
dependency and nothing to keep version-matched.
The cost, documented in the README: Sentry issues carry no trace id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ♻️ refactor(observability): fold the space span attribute into the base
AbstractSpaceMemberUseCase and AbstractSpaceAdminUseCase both extend
AbstractMemberUseCase and neither extends the other, so the spaceId span
attribute lived as two verbatim overrides, comments included — the shape that
drifts the moment one is edited. AbstractMemberUseCase now reads spaceId off
the command itself, and both overrides are gone. The existing specs assert the
attributes and are unchanged, which is what shows the behaviour is identical.
Also widen instrumentUseCases.arch.spec.ts to apps/api/src. It builds no use
cases today, so the rule is not currently violated there — the point is that it
cannot start to without the guard noticing.
Declare @opentelemetry/sdk-node as a devDependency: two specs import it and it
was resolving only through the workspace-root hoist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 49d9c2b commit 135d63c
57 files changed
Lines changed: 4342 additions & 72 deletions
File tree
- apps/api
- src
- dockerfile/prod
- docker/otel
- grafana
- dashboards
- packages
- accounts/src
- application
- adapter
- services
- infra/repositories
- coding-agent/src
- application
- adapter
- services
- infra/repositories
- commands/src
- application
- adapter
- services
- infra/repositories
- deployments/src
- application
- adapter
- services
- infra/repositories
- git/src
- application
- adapter
- infra
- jobs
- repositories
- llm/src/application/adapter
- logger/src
- node-utils
- src
- adapter
- application
- observability
- repositories
- playbook-change-applier/src
- skills/src
- application
- adapter
- services
- infra/repositories
- spaces/src
- application
- adapters
- services
- infra/repositories
- standards/src
- application
- adapter
- services
- infra/repositories
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
51 | 55 | | |
52 | 56 | | |
53 | 57 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
22 | 32 | | |
23 | 33 | | |
24 | 34 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
1 | 5 | | |
2 | 6 | | |
3 | 7 | | |
4 | 8 | | |
5 | 9 | | |
6 | 10 | | |
7 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
8 | 14 | | |
9 | 15 | | |
10 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
11 | 37 | | |
12 | 38 | | |
13 | 39 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| |||
185 | 186 | | |
186 | 187 | | |
187 | 188 | | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
188 | 193 | | |
189 | 194 | | |
190 | | - | |
| 195 | + | |
191 | 196 | | |
192 | 197 | | |
193 | 198 | | |
194 | 199 | | |
195 | 200 | | |
196 | 201 | | |
197 | 202 | | |
| 203 | + | |
| 204 | + | |
198 | 205 | | |
199 | 206 | | |
200 | 207 | | |
| |||
211 | 218 | | |
212 | 219 | | |
213 | 220 | | |
214 | | - | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
215 | 225 | | |
216 | 226 | | |
217 | 227 | | |
| |||
221 | 231 | | |
222 | 232 | | |
223 | 233 | | |
224 | | - | |
| 234 | + | |
225 | 235 | | |
226 | 236 | | |
227 | 237 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
0 commit comments