Skip to content

test: surface skipped and todo tests at the end of every run - #1587

Closed
quietbits wants to merge 2 commits into
v17-feature-branchfrom
surface-skipped-tests
Closed

test: surface skipped and todo tests at the end of every run#1587
quietbits wants to merge 2 commits into
v17-feature-branchfrom
surface-skipped-tests

Conversation

@quietbits

Copy link
Copy Markdown
Contributor

Problem

Vitest reports skipped tests as a single count in the summary. In a run of ~5500 passing tests, 6 skipped scrolls past unnoticed — so a test file that silently stops running doesn't fail anything and doesn't get spotted.

Change

A custom reporter (config/skipped-tests-reporter.ts) that prints an explicit block after each run:

  • every skipped and todo test, grouped by file, with ctx.skip() notes when present
  • files where nothing ran flagged <-- ENTIRE FILE
  • project labels ([chromium]/[firefox]) when more than one project is active
  • nothing at all when a run has no skips or todos

Registered in the node, browser, and e2e configs. Not added to the guides config — it inherits via mergeConfig, and adding it there would register it twice.

Deliberately advisory: it never fails a run. Whether a given skip is acceptable is a judgement call, so this only makes the cost visible.

Known limitations

  • Tests deselected by it.only land in the skipped bucket, so one .only in a large generated suite prints a long listing. Local only — CI rejects .only via allowOnly.
  • With --coverage, the coverage table prints after the block, since coverage is sequenced after onTestRunEnd.

@github-project-automation github-project-automation Bot moved this to Backlog (Not Ready) in DevX Aug 4, 2026
@quietbits
quietbits requested a balanced review from Copilot August 4, 2026 15:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an advisory Vitest reporter that surfaces skipped and todo tests after each run.

Changes:

  • Groups skipped/todo tests by file with notes and project labels.
  • Flags entirely skipped files.
  • Registers the reporter across Node, browser, e2e, and inherited guide tests.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
config/skipped-tests-reporter.ts Implements skip/todo reporting.
config/vitest.config.ts Registers the reporter for Node tests.
config/vitest.config.browser.ts Registers it for browser tests.
config/vitest.config.e2e.ts Registers it for e2e tests.
Suppressed comments (1)

config/skipped-tests-reporter.ts:128

  • showProject is derived only from projects that produced a skipped/todo record, rather than all projects active in the run. In a Chromium/Firefox run where only Firefox skips a test, this set has size 1 and the output omits [firefox], making the project-dependent skip ambiguous. Derive the active project set from testModules instead.
    const projects = new Set(
      [...skipped, ...todo].map((record) => record.project),
    );
    const showProject = projects.size > 1;

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread config/skipped-tests-reporter.ts Outdated
Comment thread config/skipped-tests-reporter.ts Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

config/skipped-tests-reporter.ts:165

  • Please add automated coverage for this reporter. Its core behavior (todo-vs-skip classification, whole-file detection across projects, project labels, ctx.skip() notes, and suppressing output when nothing is skipped) is currently untested, so a Vitest API or formatting regression could silently defeat the exact safeguard this PR introduces. A focused reporter test using representative modules or a small Vitest fixture would make these guarantees enforceable; similar config/ logic is unit-tested in test/unit/guide-snippets.test.ts.
  onTestRunEnd(testModules: ReadonlyArray<TestModule>): void {

@quietbits
quietbits requested a review from Ryang-21 August 4, 2026 17:19
@quietbits

Copy link
Copy Markdown
Contributor Author

Closing in favor of #1589

@quietbits quietbits closed this Aug 4, 2026
@github-project-automation github-project-automation Bot moved this from Backlog (Not Ready) to Done in DevX Aug 4, 2026
@quietbits
quietbits deleted the surface-skipped-tests branch August 4, 2026 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants