[Coverage audit] Risk-based mapping of the suite against where Langflow actually breaks

[daniellicnerski1]

Why

QA-CHECKLIST.md reports coverage against itself. It can say how much of it is
done; it cannot say what it never listed. This audit answers the other question —
are the tests we build covering the critical points of Langflow? — by measuring
from the product instead of from our own results.

Method and full data: docs/coverage-heatmap/ (added by the PR below).

risk = probability × impact, mitigated by coverage. Probability comes from
1912 upstream bug issues (recency-weighted, jira-weighted) plus four years
of code churn. Impact is anchored to what langflow.org advertises, not intuition.
Coverage enters as mitigation: tests do not stop Langflow breaking, they stop the
break reaching users unseen.

Verdict

7 of the 9 highest-risk areas are properly defended (mitigation 0.75–0.80).
The suite is where Langflow is critical. The misalignment is allocation: the two
areas at the matrix maximum — MCP and Agents, the two things the product is
sold as — sit at 0.67 and 0.71, below Observability and Knowledge/files, which
carry half the inherent risk at 0.80.

Residual-risk ranking (the work queue)

#	Residual	Area	Nature
1	15.0	security	no checklist entry at all
2	8.2	MCP	advertised promise, under-defended
3	7.4	Agents	advertised promise, under-defended
4	6.3	A2A	0.21 mitigation
5	6.0	Build / graph engine	subtle engine defects unasserted
6	6.0	i18n	no checklist entry at all

Follow-up items this audit produces

Each becomes its own issue and is scheduled into a wave — this issue is the
umbrella, not the work.

• security/ — 11 bullets. SSRF (ensure_url regression, langflow#14264),
  code-execution endpoints (/api/v1/validate/code, reported in 2023 and
  again in 2026 — the strongest recurrence in the corpus), secret exposure in
  traces, tweaks injection
• i18n/ — 5 bullets. Blocked on a prerequisite: CONTRIBUTING.md pins
  the browser locale to en-US for every project and requires multi-locale to
  be raised as its own parameterisation issue. That issue must land first
• deployments/ — 8 bullets. The 1.12 deploy page and publishing stepper
  (langflow#12303), 20+ components, zero coverage
• memory/ — 8 bullets. Memory Base registration; selectors already
  harvested from a live 1.12.0 instance and recorded in the spec doc
• Graph engine contract (§12.6) — cycles, partial failure, execution order
• MCP install / project config (§14.1) — the uncovered half of the surface
• Raise upstream: the Memories modal ships almost no data-testid (the
  primary action, both inputs, the search field and the empty state have none)

Findings the team should act on regardless of the wave

The [-] backlog is 41 items smaller than it looked. 37 template bullets
marked "automated, needs validation" had no automation of ours — their references
pointed at core/integrations/, which is Langflow's own upstream suite. Seven more
were listed twice. ROADMAP.md states that "the ~171 [-] promotions are what
make 50–60/wave reachable"; that premise needs revisiting before Wave 6 is
filled.

Four naming collisions hid real surface, and each cost a separate discovery
pass: template (Prompt Template component vs starter project), language
(Language Model vs locale), deployment (Azure deployment name vs the Deployments
page), memory (agent conversation memory vs Memory Base). Worth knowing before
the next audit.

Numbers

500 → 532 bullets; 76% → 72% coverage. The drop is entirely denominator — no test
was lost. [-] falls 89 → 48 for the opposite of a good reason: the count stops
crediting automation that does not exist.

Confidence

17% of the issue corpus could not be classified (325 of 1912) and is reported
rather than distributed, which is how security and i18n were found at all.
Good enough to order areas; not good enough for fine distinctions between
neighbours in the ranking.

[daniellicnerski1]

Two corrections after the initial audit, both applied on the branch.

Deployments is out of scope and has been removed. Its only implemented destination is watsonx Orchestrate (WatsonxOrchestrateDeploymentService, watsonx_orchestrate) — there is no vendor-neutral mechanism underneath the page, so it is a single vendor integration, the same class as the excluded bundles. Its apparent bug evidence was a naming collision: all 21 issues matching "deployment" are docker / k8s / Render / Railway / GCP infrastructure, already counted under docker/deploy. The page itself has zero reported defects. Strike the deployments/ item from the follow-up list above.

The other correction is mine to own: two areas were scored with invented numbers. Deployments and Memory Base were first given estimated churn and bug weights instead of measured ones. Measuring gave:

	invented	measured
deployments churn	8.0	4.25
deployments bugs	0.5	0.0
memory churn	6.0	1.02
memory bugs	0.5	4.0

Memory's churn was six times smaller than the figure used, and the invented values had put both areas at the top of the ranking — positions the measurement does not support. Every probability input is now measured; data.json records this under measuredInputs so a later reader can tell which figures were derived.

Corrected ranking (23 areas): security 15.0 · MCP 10.5 · Memory Base 8.0 · Agents 7.4 · Templates 7.4 · A2A 6.3 · graph engine 6.0 · i18n 6.0.

Auth reads 5.4 again — it briefly showed 8.1 only because quintiles are relative to the set and the set had two extra areas in it. Nothing about Auth changed.

One open question for whoever takes the Memory Base item: its three real bug reports are all about ingestion, which the 8 bullets deliberately exclude — the registration surface they cover has no reported defects yet. Widening the item to ingestion is a scope decision, not a consequence of the audit.