chore: map coverage against a probability x impact risk model - #1352
Open
daniellicnerski1 wants to merge 4 commits into
Open
chore: map coverage against a probability x impact risk model#1352daniellicnerski1 wants to merge 4 commits into
daniellicnerski1 wants to merge 4 commits into
Conversation
Audits what the suite automates against where Langflow actually breaks, and records the gaps the checklist could not see because they had no entry in it. Method (docs/coverage-heatmap/): risk = probability x impact, mitigated by coverage. Probability is measured from the product — 1912 upstream bug issues (recency-weighted) plus four years of code churn — never from our own test results, which only know where we already test. Impact is anchored to what langflow.org advertises. Full data and per-row rationale in data.json. New areas, none of which had a single bullet before: - security/ (11) SSRF, code-execution endpoints, secret exposure, tweaks - i18n/ (5) language selection and locale resilience - deployments/ (8) the 1.12 deploy page and publishing stepper - memory/ (8) Memory Base registration, selectors harvested live Also adds the graph engine's own contract (12.6) and the uncovered MCP install/project-config surface (14.1). Integrity fixes found while validating: - templates: 41 bullets -> 34. Seven were listed twice, and 37 marked [-] had no automation of ours at all — their references pointed at core/integrations/, which is Langflow's upstream suite, not this repo. Reclassified to [ ], or [~] where a template is instantiated as a fixture. - Composio marked out of scope: it is a vendor bundle, no longer supported. - Removed an orphan bullet pointing at agent-reasoning-steps.spec.ts, deleted in 1b3c686; the behaviour is covered by an @stable bullet in 6.1. Totals move 500 -> 532 bullets and 76% -> 72%. The drop is entirely the denominator: no test was lost. [-] falls 89 -> 48 for the opposite of a good reason — the count stops crediting automation that does not exist, which also invalidates the ROADMAP capacity premise that ~171 [-] promotions are available. Generated blocks are untouched and regenerate on merge.
The committed data.json was computed before the new bullets landed and went stale against the checklist in the same commit — the exact defect this audit kept finding in the checklist itself. - MCP read 25 bullets / 0.67 while the section holds 29 / 0.58 (residual 8.2 -> 10.5). The four install/project-config bullets were added after the figure was computed. - Templates read 41 / 0.42 while the corrected section holds 34 / 0.08 (residual 4.6 -> 7.4). The mitigation fix was discussed but never regenerated. - Deployments and Memory Base were missing from the matrix entirely; they were discovered after the first generation. Now scored: both 0.00 mitigation. - Build / graph engine keeps 0.70: its mitigation comes from indirect detection (63 @stable specs trigger a run), which does not depend on the bullet count. Mitigation is now DERIVED from the live bullet states rather than carried over, and data.json records that, so the next regeneration cannot silently diverge. Two consequences worth reading before the ranking: Adding two areas re-scores every quintile, because quintiles are relative to the set. Auth moved 5.4 -> 8.1 for that reason alone; nothing about Auth changed. The top three are now security (15.0), deployments (15.0) and memory (12.0) — all at 0.00 mitigation, all for the same reason: no bullet existed, so they were invisible to every coverage figure including the 76% headline. The first area with real coverage appears at rank 4.
Two corrections, one of which is a process failure worth recording. Deployments is out of scope. Its only implemented destination is watsonx Orchestrate (WatsonxOrchestrateDeploymentService, watsonx_orchestrate) — there is no vendor-neutral mechanism underneath the page, so it is a single vendor integration, the same class as the excluded bundles. Its apparent bug evidence was a naming collision: all 21 issues matching "deployment" are docker / k8s / Render / Railway / GCP infrastructure, already counted under docker/deploy. The page itself has zero reported defects. Section and MODULES entry removed. The rest of this commit exists because I invented numbers. Deployments and Memory Base were first scored with estimated churn and bug weights (8.0/6.0 and 0.5/0.5) instead of measured ones. Measuring gave: deployments churn 4.25 (not 8.0) bugs 0.0 (not 0.5) memory churn 1.02 (not 6.0) bugs 4.0 (not 0.5) Memory's churn was six times smaller than the figure I used, and the invented values had placed both areas at the top of the ranking — positions measurement does not support. Every probability input in data.json is now measured, and `measuredInputs` records that so the next reader can tell the difference. Memory Base keeps its 8 bullets and lands at rank 3 (residual 8.0) on the measured figures. Worth noting for whoever picks it up: its three real bug reports are all about *ingestion*, which these bullets deliberately exclude — the registration surface they do cover has no reported defects yet. Ranking after the correction: security 15.0, MCP 10.5, Memory Base 8.0, Agents 7.4, Templates 7.4. Auth returns to 5.4 — it had read 8.1 only because quintiles are relative and the set had two extra areas in it.
Team decision (2026-08-07): registration and ingestion are one product surface
but two pieces of work. §20.1-20.3 keep registration; §20.4 adds ingestion as
the second item, 7 bullets.
The split follows the evidence, and it points away from the bullets written
first: all three of Memory Base's real upstream defects are ingestion —
#13884 (chunk settings not applied, jira), #13883 (unreachable Ollama, jira)
and #12277 (Google embedding models rejected) — while the registration surface
§20.1-20.3 cover has no reported defect yet.
Routes confirmed against the running 1.12.0 instance, not assumed: ingest,
preview-chunks, chunks, runs, runs/{id}, cancel, connectors, test-connection.
`GET /connectors` returns exactly one connector today — `folder`, with
requires_credentials false — so a test needs no external service, only a
server-side path the instance can read.
One bullet covers something nothing asserts today: the API declares
`_check_memory_base_association` on five routes, and no test proves the guard
refuses what it is there to refuse.
Every issue number in the new areas was verified against the bug corpus before
committing. One was wrong on the first write — #13351, which does not exist;
the real one is #13884.
Memory Base stays at residual 8.0: the added bullets are all `[ ]`, so
mitigation remains 0.00 and only the bullet count moves, 8 -> 15. data.json
regenerated from the checklist so the two cannot drift.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1350
What this adds
A risk-based audit of what the suite automates against where Langflow actually
breaks, plus the checklist entries for the gaps it found. No test code — the
work items land as bullets and are scheduled into a wave.
docs/coverage-heatmap/README.md— method, ranking, verdict, declared limitsdocs/coverage-heatmap/data.json— 22 areas, every judged value with its rationaleQA-CHECKLIST.md— four new areas, two existing ones extended, three integrity fixesscripts/coverage-summary.ts— the four new module sections registered inMODULESdocs/core-functionality/memory/memory-base-registration.md— spec doc for theMemory Base item, with selectors harvested from a live 1.12.0 instance so whoever
picks it up does not repeat the scouting
Method, briefly
risk = probability × impact, mitigated by coverage. Probability is measured fromthe product — 1912 upstream
bugissues (recency-weighted) plus four years ofchurn — never from our own test results, which only know where we already test.
Impact is anchored to what langflow.org advertises. Coverage enters as mitigation,
not as a term in the sum: tests do not stop Langflow breaking, they stop a break
reaching users unseen.
New checklist areas — none had a single bullet before
security/ensure_url, langflow#14264);/api/v1/validate/codereported in 2023 and 2026; secret exposure in traces; tweaks injectioni18n/deployments/memory/Plus the graph engine's own contract (§12.6) and the uncovered MCP install /
project-config surface (§14.1).
i18n/carries a documented blocker.CONTRIBUTING.md→ Browser locale ispinned to
en-USfixes the locale for every project and requires multi-locale tobe raised as its own parameterisation issue. §18.2 cannot run under that pin, so
the section header says so rather than inviting someone to edit the shared default.
Integrity fixes found while validating
[-]had noautomation of ours — their references pointed at
core/integrations/, which isLangflow's upstream suite, not this repo. Reclassified to
[ ], or[~]where the template is instantiated as a fixture by other specs.
agent-reasoning-steps.spec.ts, deletedin
1b3c686; the behaviour is covered by an@stablebullet in §6.1.Impact on the numbers, stated plainly
500 → 532 bullets, 76% → 72% coverage. The drop is entirely denominator —
no test was lost.
[-]falls 89 → 48 for the opposite of a good reason: thecount stops crediting automation that does not exist.
This invalidates a ROADMAP premise.
ROADMAP.mdstates the "~171[-]promotions are what make 50–60/wave reachable". At least 37 of those do not exist,
so the capacity band needs revisiting before Wave 6 is filled.
Validation
No
.spec.tswas added or changed, so the 5-step test checklist does not apply.What was run:
npx tsc --noEmit— cleannpx eslint scripts/coverage-summary.ts— cleannode scripts/check-checklist-guard.mjs— no generated block editednpx ts-node scripts/check-checklist-coverage.ts— every@stable/documented specis referenced by a bullet
npm run test:units— 442 passed, 0 failednpm run test:scripts— 712 passed, 0 failedMODULESaddition was verified by runningnpm run coverage:summarylocallyand confirming the four new rows render — the output was reverted and is not in
this PR (issue Guard QA-CHECKLIST generated blocks from PR edits (stop recurring merge conflicts) #741; the generated blocks regenerate on merge)
Live scouting for the Memory Base spec doc ran against a real Langflow 1.12.0
instance; every selector in it came from the DOM, none was invented.
Confidence and limits
17% of the issue corpus (325 of 1912) could not be classified and is reported
rather than distributed — which is how
securityandi18nsurfaced at all. Theranking is good enough to order areas, not to separate neighbours. Fragility and
impact are judgement, one rationale per row in
data.json. Three of my ownpredictions were refuted by the data during the audit and are recorded in the README.