Skip to content

Commit f9789e3

Browse files
Copilot CICopilot
andcommitted
Stage the replication candidate before the fix phase can time out
Seven of the eight timed-out runs in the cached corpus had already reproduced their issue, authored a failing test and cleared the negative control. All seven published nothing, because the candidate manifest was written after the fix phase and a task timeout kills the process rather than raising into the try/catch around it. Extract the manifest into a single $writeCandidateManifest scriptblock and call it twice: before the fix phase to stage the reproduction, and after it to announce READY with any fix attached. One writer, so the two manifests cannot drift apart. The pipeline guard that cross-checks the gate allowlist against the fields the orchestrator writes matched top-level keys by a hard-coded eight-space indent, so the move into a scriptblock emptied its derived list. Derive the indent from the manifest instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 735ac9a2-7bec-4baa-ad19-c298e5bc795a
1 parent 341f948 commit f9789e3

3 files changed

Lines changed: 149 additions & 54 deletions

File tree

.github/scripts/Replicate-Issue.Tests.ps1

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13439,3 +13439,59 @@ Describe 'An attempt classifier logs the text it classified' {
1343913439
'veto that cannot be attributed to an attempt cannot be audited')
1344013440
}
1344113441
}
13442+
13443+
Describe 'A verified reproduction survives a fix-phase timeout' {
13444+
# Run 15089945 reproduced its issue, passed verification, cleared the
13445+
# negative control and selected a winning fix - then hit
13446+
# "##[error]The task has timed out." before the candidate manifest existed,
13447+
# so the publisher reported "No replication candidate manifest was
13448+
# produced; nothing to validate." and opened nothing. 7 of the 8 timeouts
13449+
# in the cached corpus had already passed verification. A task timeout
13450+
# kills the process, so the existing try/catch around the fix phase cannot
13451+
# help: the manifest has to be on disk before the panel starts.
13452+
BeforeAll {
13453+
$script:ReplSource = Get-Content -Raw -LiteralPath (
13454+
Join-Path $PSScriptRoot 'Replicate-Issue.ps1')
13455+
$script:StageAt = $script:ReplSource.IndexOf('& $writeCandidateManifest $false')
13456+
$script:FixAt = $script:ReplSource.IndexOf('$fixOutcome = Invoke-ReplicationFixPhase')
13457+
$script:FinalAt = $script:ReplSource.IndexOf('& $writeCandidateManifest $true')
13458+
$script:DefAt = $script:ReplSource.IndexOf('$writeCandidateManifest = {')
13459+
}
13460+
13461+
It 'writes the manifest before the fix phase starts' {
13462+
$script:StageAt | Should -BeGreaterThan 0
13463+
$script:FixAt | Should -BeGreaterThan 0
13464+
$script:StageAt | Should -BeLessThan $script:FixAt
13465+
}
13466+
13467+
It 'defines the writer above the call that stages it' {
13468+
# All four sites are top level, so source order is execution order.
13469+
$script:DefAt | Should -BeGreaterThan 0
13470+
$script:DefAt | Should -BeLessThan $script:StageAt
13471+
}
13472+
13473+
It 'writes it again after the fix phase, so a fix is not lost' {
13474+
$script:FinalAt | Should -BeGreaterThan $script:FixAt
13475+
}
13476+
13477+
It 'announces READY only once, after the fix phase' {
13478+
# The staged write must not claim READY: that marker is what the
13479+
# corpus census and the publisher key on.
13480+
([regex]::Matches($script:ReplSource,
13481+
'ISSUE REPLICATION CANDIDATE READY')).Count | Should -Be 1
13482+
$readyAt = $script:ReplSource.IndexOf('ISSUE REPLICATION CANDIDATE READY')
13483+
$script:ReplSource.Substring($readyAt - 400, 400) | Should -Match '\$Announce'
13484+
}
13485+
13486+
It 'still records a staged manifest so the timeout case stays measurable' {
13487+
$script:ReplSource | Should -Match 'ISSUE REPLICATION CANDIDATE STAGED'
13488+
}
13489+
13490+
It 'stages the identical manifest the fix-less run would publish' {
13491+
# Two hand-written manifests would drift, and the drift would only ever
13492+
# show up as a malformed candidate on the timeout path.
13493+
([regex]::Matches($script:ReplSource,
13494+
'\$writeCandidateManifest = \{')).Count | Should -Be 1
13495+
}
13496+
}
13497+

.github/scripts/Replicate-Issue.ps1

Lines changed: 81 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -8137,10 +8137,89 @@ Explain in lighterTypesRejected why the previous tier could not observe it. Choo
81378137

81388138
New-TestPatch -Files $generatedFiles
81398139

8140+
# A task timeout kills this process outright, so the guard above -
8141+
# which only survives the fix phase *failing* - cannot help: run 15089945
8142+
# reproduced its issue, verified it, cleared the negative control and
8143+
# picked a winning fix, then timed out before this manifest existed. The
8144+
# publisher then said "No replication candidate manifest was produced;
8145+
# nothing to validate." 7 of the 8 timeouts in the cached corpus had
8146+
# already passed verification, which makes this the most expensive way
8147+
# the pipeline loses work.
8148+
#
8149+
# Writing the manifest once before the fix phase and again after leaves a
8150+
# valid reproduction-only candidate on disk the whole time the fix panel
8151+
# runs. The artifact upload is succeededOrFailed, so it survives, and the
8152+
# manifest already models a fix-less candidate (fixFiles = @()).
8153+
$writeCandidateManifest = {
8154+
param([bool]$Announce)
8155+
# Replacing newlines after truncation left a trailing space, and the gate
8156+
# rejects an untrimmed manifest step, so build 15030804 reproduced its issue
8157+
# and was discarded for whitespace. Collapse and trim after the replacement.
8158+
$reproductionSteps = @($testProposal.reproductionSteps | ForEach-Object {
8159+
([regex]::Replace(
8160+
((ConvertTo-ReplicationSafeLog $_ 300) -replace '\r|\n', ' '),
8161+
'\s+',
8162+
' ')).Trim()
8163+
} | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | Select-Object -First 10)
8164+
[ordered]@{
8165+
schemaVersion = 1
8166+
issueNumber = $IssueNumber
8167+
platform = $Platform
8168+
baseSha = $BaseSha.ToLowerInvariant()
8169+
status = 'reproduced'
8170+
blocked = $null
8171+
selectedDevice = [ordered]@{
8172+
id = $selectedDeviceId
8173+
name = $DeviceName
8174+
osVersion = $DeviceOSVersion
8175+
}
8176+
attempts = [ordered]@{
8177+
sandbox = $sandboxAttempts
8178+
automatedTest = $testAttempts
8179+
}
8180+
reproductionSteps = $reproductionSteps
8181+
expectedBehavior = ConvertTo-ReplicationSafeLog ([string]$testProposal.expectedBehavior) 500
8182+
observedBehavior = ConvertTo-ReplicationSafeLog ([string]$testProposal.observedBehavior) 500
8183+
testType = [string]$testProposal.testType
8184+
testFilter = [string]$testProposal.testFilter
8185+
testClassName = [string]$verifierMetadata.ClassName
8186+
testMethodName = [string]$verifierMetadata.MethodName
8187+
expectedFailureSignature = [string]$testProposal.expectedFailureSignature
8188+
files = $generatedFiles
8189+
sandboxFiles = [ordered]@{
8190+
xaml = 'sandbox/MainPage.xaml'
8191+
codeBehind = 'sandbox/MainPage.xaml.cs'
8192+
appiumPlan = 'sandbox/appium-plan.json'
8193+
}
8194+
reproductionResult = 'reproduction-result.json'
8195+
evidenceManifest = 'evidence/evidence.json'
8196+
verificationResult = 'verification/verification-result.json'
8197+
negativeControl = $negativeControl
8198+
patch = 'test.patch'
8199+
fixFiles = if ($fixOutcome) { @($fixOutcome.Files) } else { @() }
8200+
fixPatch = if ($fixOutcome) { 'fix.patch' } else { $null }
8201+
fixRootCause = if ($fixOutcome) { $fixOutcome.RootCause } else { $null }
8202+
fixApproach = if ($fixOutcome) { $fixOutcome.Approach } else { $null }
8203+
fixRejectedApproaches = if ($fixOutcome) { @($fixOutcome.RejectedApproaches) } else { @() }
8204+
} | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $candidatePath -Encoding utf8NoBOM
8205+
8206+
if ($Announce) {
8207+
Write-Host "ISSUE REPLICATION CANDIDATE READY: $candidatePath"
8208+
} else {
8209+
Write-Host "ISSUE REPLICATION CANDIDATE STAGED: $candidatePath"
8210+
}
8211+
}
8212+
8213+
8214+
# Stage the reproduction before the fix phase can time out. This writes the
8215+
# same manifest the run would publish with no fix, so a process killed
8216+
# during the panel leaves exactly what a fix-less run leaves.
8217+
$fixOutcome = $null
8218+
& $writeCandidateManifest $false
8219+
81408220
# The reproduction is certified at this point and its patch is already
81418221
# written, so nothing the fix phase does can reach it. Any failure inside
81428222
# returns $null and publishes the reproduction alone.
8143-
$fixOutcome = $null
81448223
if ($negativeControl) {
81458224
try {
81468225
$fixOutcome = Invoke-ReplicationFixPhase `
@@ -8162,58 +8241,7 @@ Explain in lighterTypesRejected why the previous tier could not observe it. Choo
81628241
Remove-Item -LiteralPath $fixPatchPath -Force -ErrorAction SilentlyContinue
81638242
}
81648243

8165-
# Replacing newlines after truncation left a trailing space, and the gate
8166-
# rejects an untrimmed manifest step, so build 15030804 reproduced its issue
8167-
# and was discarded for whitespace. Collapse and trim after the replacement.
8168-
$reproductionSteps = @($testProposal.reproductionSteps | ForEach-Object {
8169-
([regex]::Replace(
8170-
((ConvertTo-ReplicationSafeLog $_ 300) -replace '\r|\n', ' '),
8171-
'\s+',
8172-
' ')).Trim()
8173-
} | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | Select-Object -First 10)
8174-
[ordered]@{
8175-
schemaVersion = 1
8176-
issueNumber = $IssueNumber
8177-
platform = $Platform
8178-
baseSha = $BaseSha.ToLowerInvariant()
8179-
status = 'reproduced'
8180-
blocked = $null
8181-
selectedDevice = [ordered]@{
8182-
id = $selectedDeviceId
8183-
name = $DeviceName
8184-
osVersion = $DeviceOSVersion
8185-
}
8186-
attempts = [ordered]@{
8187-
sandbox = $sandboxAttempts
8188-
automatedTest = $testAttempts
8189-
}
8190-
reproductionSteps = $reproductionSteps
8191-
expectedBehavior = ConvertTo-ReplicationSafeLog ([string]$testProposal.expectedBehavior) 500
8192-
observedBehavior = ConvertTo-ReplicationSafeLog ([string]$testProposal.observedBehavior) 500
8193-
testType = [string]$testProposal.testType
8194-
testFilter = [string]$testProposal.testFilter
8195-
testClassName = [string]$verifierMetadata.ClassName
8196-
testMethodName = [string]$verifierMetadata.MethodName
8197-
expectedFailureSignature = [string]$testProposal.expectedFailureSignature
8198-
files = $generatedFiles
8199-
sandboxFiles = [ordered]@{
8200-
xaml = 'sandbox/MainPage.xaml'
8201-
codeBehind = 'sandbox/MainPage.xaml.cs'
8202-
appiumPlan = 'sandbox/appium-plan.json'
8203-
}
8204-
reproductionResult = 'reproduction-result.json'
8205-
evidenceManifest = 'evidence/evidence.json'
8206-
verificationResult = 'verification/verification-result.json'
8207-
negativeControl = $negativeControl
8208-
patch = 'test.patch'
8209-
fixFiles = if ($fixOutcome) { @($fixOutcome.Files) } else { @() }
8210-
fixPatch = if ($fixOutcome) { 'fix.patch' } else { $null }
8211-
fixRootCause = if ($fixOutcome) { $fixOutcome.RootCause } else { $null }
8212-
fixApproach = if ($fixOutcome) { $fixOutcome.Approach } else { $null }
8213-
fixRejectedApproaches = if ($fixOutcome) { @($fixOutcome.RejectedApproaches) } else { @() }
8214-
} | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $candidatePath -Encoding utf8NoBOM
8215-
8216-
Write-Host "ISSUE REPLICATION CANDIDATE READY: $candidatePath"
8244+
& $writeCandidateManifest $true
82178245
}
82188246
catch {
82198247
$rawReason = [string]$_.Exception.Message

.github/scripts/Replication-Pipeline.Tests.ps1

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -590,8 +590,19 @@ Describe 'The trusted publisher stages every module its gate loads' {
590590

591591
# Only the manifest's own top-level keys are validated by the gate's
592592
# allowlist; nested hashtable keys are checked by their own rules.
593+
# Derive the top-level indent rather than hard-coding it: the manifest
594+
# moved into a scriptblock so it could be written before the fix phase,
595+
# which shifted every key four columns and silently emptied this list.
596+
$indent = [regex]::Match(
597+
$manifestBlock, "(?m)^([ ]*)schemaVersion = 1").Groups[1].Value
598+
if (-not $indent) {
599+
$lineStart = $orchestrator.LastIndexOf("`n", $manifestStart)
600+
$indent = $orchestrator.Substring(
601+
$lineStart + 1, $manifestStart - $lineStart - 1)
602+
}
603+
$indent | Should -Match '^ +$'
593604
$written = @([regex]::Matches(
594-
$manifestBlock, "(?m)^ ([A-Za-z][A-Za-z0-9]*) = ") |
605+
$manifestBlock, "(?m)^$indent([A-Za-z][A-Za-z0-9]*) = ") |
595606
ForEach-Object { $_.Groups[1].Value })
596607
$written | Should -Contain 'negativeControl'
597608
$written.Count | Should -BeGreaterThan 10

0 commit comments

Comments
 (0)