Skip to content

Frontend line-coverage floor is nondeterministic on CI and can fail spuriously #1889

Description

@giswqs

Summary

npm run test:frontend:coverage gates on --test-coverage-lines=78. That line number is not reproducible on CI: two runs over byte-identical sources produced 81.82% and 76.47%, and the second is below the floor. Branch and function coverage were identical in both.

This means the line floor can redden a PR that changed nothing relevant. It already did: PR #1887 (a release version bump plus docs, zero source files touched) failed with

Error: 76.47% line coverage does not meet threshold of 78%.

while main's own run of the same sources reported 81.82% and did not.

Evidence

Two CI runs, same TypeScript sources (the PR differed only in package.json versions, docs, and lockfiles):

run files counted lines branches functions
main push (run 31764040051) 444 81.82% 84.30% 60.36%
PR #1887 (run 31764356629) 444 76.47% 84.30% 60.36%

Both executed identically: 5935 tests, 1297 suites, 5934 pass, 0 fail, 1 skipped, 0 todo.

Comparing the two per-file tables by full path:

  • 114 of 444 files report different line coverage
  • 0 files report different function coverage
  • 0 files report different branch coverage
  • the counted file set is identical (no file entered or left the report)

The swings are large, not rounding noise:

file run A run B
packages/core/src/storymap-sample.ts 99.21% 34.65%
packages/map/src/qml-import.ts 97.53% 48.48%
packages/map/src/qml-export.ts 94.38% 44.52%
packages/core/src/visibility.ts 93.42% 46.05%
packages/map/src/sld-export.ts 97.70% 54.75%
packages/core/src/layer-ref-scrub.ts 97.96% 57.14%

Affected files cluster in packages/plugins (46), packages/core (37), packages/map (16), apps/geolibre-desktop (9). Some are referenced by many test files (visibility.ts by 33); others (storymap-sample.ts, layer-ref-scrub.ts) by none directly, reaching the report only transitively through a package index.

Not reproducible locally

Two consecutive local runs on one tree differed in 1 file (packages/core/src/store.ts, 75.35% vs 76.36%), totals 82.84% vs 82.86%. So this looks specific to the CI runner, plausibly node --test's worker-process count and the merging of per-process V8 coverage under a 4-core, memory-constrained runner. That functions and branches are perfectly stable while lines are not suggests the line ranges, rather than the coverage data as a whole, are what get lost or partially merged.

Why it matters

Per CLAUDE.md the floors are a ratchet meant to catch real regressions. A gate that swings 5+ points run to run on identical code cannot do that: it produces failures nobody can act on, and it trains people to re-run CI until it passes, which is exactly how a genuine regression slips through.

Options

  1. Lower the line floor to sit under the observed trough (the swing bottomed at 76.47%, so ~74) and keep functions and branches where they are, since those are stable.
  2. Drop the line gate and ratchet on functions and branches only, which are reproducible run to run.
  3. Pin node --test concurrency in CI (--test-concurrency=1 or a small fixed number) and re-measure. Slower, but if it makes lines deterministic the floor can stay meaningful.
  4. Report upstream to Node if a minimal reproduction can be extracted.

I lean toward 3 for a measurement, then 1 or 2 depending on what it shows.

Context

Found while diagnosing the separate, real coverage regression fixed in #1888. That one was deterministic (function coverage, 72.90% to 60.36%) and is unrelated to this flake.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions