Skip to content

Commit 1a645b1

Browse files
Copilot CICopilot
andcommitted
A cleared control was announced by a line that called it a failure
The banner refuses to say "PASSED" when a negative control fails, for the reason its own comment gives: it would tell a reader the exact opposite of what was measured. The per-test loop eight lines above it was never given the same treatment, so a refuted control printed [UITest] Issue31059: FAILED (expected) <- green CONTROL DID NOT CLEAR <- red one after the other, and only the second was true. In the control arm a failing test is the refutation, not the expectation. Both directions are fixed, because a change to the failing branch alone leaves a cleared control reported as "PASSED (should fail!)" - the same inversion the other way round. The reproduction arm's wording is byte-identical: Validate-ReplicationCandidate.Tests.ps1 builds a console fixture from that exact string, and the markdown report is deliberately untouched so existing validators keep reading what they read today. The full-verification renderers were checked and left alone; their arms already label themselves correctly. Mutation tested: reverting the change fails 3 of 5, making only the failing branch arm-aware fails 2, drifting the reproduction wording fails 1. Suites 1731 passed, 2 failed, both pre-existing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
1 parent ef1ffc0 commit 1a645b1

2 files changed

Lines changed: 67 additions & 2 deletions

File tree

.github/skills/verify-tests-fail-without-fix/scripts/Verify-TestsFail.Tests.ps1

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1801,3 +1801,54 @@ Describe 'The authoritative test result is kept as evidence' {
18011801
ConvertFrom-Json).resultFile | Should -BeExactly ''
18021802
}
18031803
}
1804+
1805+
Describe 'The per-test line knows which control arm it is reporting' {
1806+
# The banner already refuses to say "PASSED" for a failing negative control,
1807+
# for the reason its own comment gives: it would tell a reader the exact
1808+
# opposite of what was measured. The per-test loop a few lines above it was
1809+
# not given the same treatment, so build 15085903 printed
1810+
# [UITest] Issue31059: FAILED (expected) <- green
1811+
# CONTROL DID NOT CLEAR <- red
1812+
# one after the other, and only the second was true.
1813+
BeforeAll {
1814+
$script:PerTestBlock = $script:VerifierSource.Substring(
1815+
$script:VerifierSource.IndexOf('# Show per-test results'),
1816+
2200)
1817+
}
1818+
1819+
It 'reports a failing negative control as the refutation, not as the expectation' {
1820+
$script:PerTestBlock | Should -Match 'control did not clear'
1821+
}
1822+
1823+
It 'reports a passing negative control as the expected outcome' {
1824+
$script:PerTestBlock | Should -Match 'expected with the trigger removed'
1825+
}
1826+
1827+
It 'branches on Purpose for both the failing and the passing case' {
1828+
# Guarding both halves: a fix applied to only the failing branch leaves a
1829+
# passing control reported as "PASSED (should fail!)", which is the same
1830+
# inversion in the other direction.
1831+
([regex]::Matches($script:PerTestBlock, "Purpose -eq 'NegativeControl'")).Count |
1832+
Should -BeGreaterOrEqual 2
1833+
}
1834+
1835+
It 'leaves the reproduction arm wording byte-identical' {
1836+
# Validate-ReplicationCandidate.Tests.ps1 builds a console fixture
1837+
# containing this exact string, and the report renderer is deliberately
1838+
# untouched so existing validators keep reading what they read today.
1839+
$script:PerTestBlock | Should -Match 'FAILED \u2705 \(expected\)'
1840+
$script:PerTestBlock | Should -Match 'PASSED \u274c \(should fail!\)'
1841+
}
1842+
1843+
It 'still classifies env and build errors before either verdict' {
1844+
# An infrastructure failure is neither arm's result, and it has to keep
1845+
# outranking both so a build break cannot be read as a cleared control.
1846+
$envAt = $script:PerTestBlock.IndexOf('ENV ERROR')
1847+
$buildAt = $script:PerTestBlock.IndexOf('BUILD ERROR')
1848+
$verdict = $script:PerTestBlock.IndexOf("Purpose -eq 'NegativeControl'")
1849+
$envAt | Should -BeGreaterThan 0
1850+
$buildAt | Should -BeGreaterThan 0
1851+
$envAt | Should -BeLessThan $verdict
1852+
$buildAt | Should -BeLessThan $verdict
1853+
}
1854+
}

.github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2373,9 +2373,23 @@ if ($DetectedFixFiles.Count -eq 0) {
23732373
} elseif ($r.Error) {
23742374
Write-Host " $icon [$($r.TestType)] $($r.TestName): ⚠️ ERROR — $($r.Error)" -ForegroundColor Yellow
23752375
} elseif (-not $r.Passed) {
2376-
Write-Host " $icon [$($r.TestType)] $($r.TestName): FAILED ✅ (expected)" -ForegroundColor Green
2376+
if ($Purpose -eq 'NegativeControl') {
2377+
# A failing test is the expected result for the reproduction arm
2378+
# and the refutation for this one. Printing the reproduction's
2379+
# green "(expected)" here told a reader the exact opposite of
2380+
# what was measured, directly above the CONTROL DID NOT CLEAR
2381+
# banner that says so - the same mistake the banner below was
2382+
# already written to avoid.
2383+
Write-Host " $icon [$($r.TestType)] $($r.TestName): FAILED ❌ (control did not clear)" -ForegroundColor Red
2384+
} else {
2385+
Write-Host " $icon [$($r.TestType)] $($r.TestName): FAILED ✅ (expected)" -ForegroundColor Green
2386+
}
23772387
} else {
2378-
Write-Host " $icon [$($r.TestType)] $($r.TestName): PASSED ❌ (should fail!)" -ForegroundColor Red
2388+
if ($Purpose -eq 'NegativeControl') {
2389+
Write-Host " $icon [$($r.TestType)] $($r.TestName): PASSED ✅ (expected with the trigger removed)" -ForegroundColor Green
2390+
} else {
2391+
Write-Host " $icon [$($r.TestType)] $($r.TestName): PASSED ❌ (should fail!)" -ForegroundColor Red
2392+
}
23792393
}
23802394
}
23812395
Write-Host ""

0 commit comments

Comments
 (0)