Skip to content

Commit a43de9d

Browse files
astaykovastaykov
authored
Implement error sanitization and reporting improvements in tests (#1426)
Fix #1422 This pull request introduces a comprehensive overhaul of error sanitization for PowerShell test diagnostics to prevent sensitive credential data from being persisted in logs, reports, or error records. The main improvements include a new defense-in-depth sanitization function, safe error record/message generation, and updated error formatting and logging logic to ensure only allow-listed diagnostic fields are exposed. Extensive tests are added to validate these protections. **Error sanitization and redaction:** * Added `Protect-ZtReportText` function to redact credential-bearing HTTP headers and sensitive values from report content before persistence, and integrated it into `Add-ZtTestResultDetail`. * Updated test result persistence and logging to always sanitize error details, including at the TestResult persistence boundary and in optional test logs. **Safe error record and message generation:** * Introduced `New-ZtSafeErrorRecord` to create sanitized error records with only allow-listed diagnostic fields and without unsafe TargetObject data. This prevents request/response objects containing bearer tokens from being retained in test statistics or passed into logging. * Added `Get-ZtSafeErrorMessage` to extract safe, structured diagnostic summaries (request method/path, HTTP status, Graph error code, and correlation IDs) from error records, omitting unstructured exception messages, headers, and response bodies. **Error formatting and reporting:** * Implemented `Format-ZtTestErrorDetail` to generate markdown-formatted error summaries using only sanitized fields for failed tests. It adds bounded metadata such as exception type, error ID, HTTP status, and source location. * Refactored error handling in `Invoke-ZtTest.ps1` and `Write-ZtTestError.ps1` to use the new safe error record/message pipeline, ensuring all error reporting is sanitized and consistent. **Testing:** * Added `ErrorSanitization.Tests.ps1` with comprehensive tests to verify that credentials and sensitive values are never persisted in logs or reports, and that only safe diagnostic information is exposed.
2 parents c28b05a + 5d95210 commit a43de9d

8 files changed

Lines changed: 327 additions & 20 deletions

File tree

‎code-tests/commands/ErrorSanitization.Tests.ps1‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
Describe "Error sanitization" {
2+
BeforeAll {
3+
$here = $PSScriptRoot
4+
$srcRoot = Join-Path $here "../../src/powershell"
5+
6+
. (Join-Path $srcRoot "private/core/Protect-ZtReportText.ps1")
7+
. (Join-Path $srcRoot "private/core/Get-ZtHttpStatusCode.ps1")
8+
. (Join-Path $srcRoot "private/core/Get-ZtTestStatus.ps1")
9+
. (Join-Path $srcRoot "private/tests/Get-ZtSafeErrorMessage.ps1")
10+
. (Join-Path $srcRoot "private/tests/New-ZtSafeErrorRecord.ps1")
11+
. (Join-Path $srcRoot "private/tests/Format-ZtTestErrorDetail.ps1")
12+
. (Join-Path $srcRoot "private/tests/Write-ZtTestError.ps1")
13+
. (Join-Path $srcRoot "private/tests/Write-ZtTestLog.ps1")
14+
. (Join-Path $srcRoot "private/core/Add-ZtTestResultDetail.ps1")
15+
16+
function New-CanaryErrorRecord {
17+
$script:canaryToken = 'eyJhbGciOiJub25lIn0.eyJhcHBpZCI6InJlZGFjdGlvbi10ZXN0In0.signature'
18+
$request = [System.Net.Http.HttpRequestMessage]::new([System.Net.Http.HttpMethod]::Get, "https://graph.microsoft.com/beta/example?access_token=$script:canaryToken")
19+
$request.Headers.Authorization = [System.Net.Http.Headers.AuthenticationHeaderValue]::new('Bearer', $script:canaryToken)
20+
$exception = [System.Exception]::new(@"
21+
Exception calling ""InvokeGlobal"" with ""1"" argument(s): ""GET https://graph.microsoft.com/beta/example?access_token=$script:canaryToken
22+
HTTP/1.1 401 Unauthorized
23+
request-id: 11111111-1111-1111-1111-111111111111
24+
client-request-id: 22222222-2222-2222-2222-222222222222
25+
Authorization: Bearer $script:canaryToken
26+
Cookie: session=example-cookie
27+
{"error":{"code":"UnknownError","message":"example response body"}}
28+
"@)
29+
30+
return [System.Management.Automation.ErrorRecord]::new(
31+
$exception,
32+
'GraphRequestFailed',
33+
[System.Management.Automation.ErrorCategory]::PermissionDenied,
34+
$request
35+
)
36+
}
37+
}
38+
39+
BeforeEach {
40+
$script:__ZtSession = @{
41+
TestResultDetail = [PSCustomObject]@{ Value = @{} }
42+
}
43+
$script:loggedErrorRecord = $null
44+
$script:addedResult = $null
45+
46+
function Write-PSFMessage {
47+
param($Level, $Message, $StringValues, $Target, $ErrorRecord, $Tag)
48+
$script:loggedErrorRecord = $ErrorRecord
49+
}
50+
51+
function Update-ZtProgressState {}
52+
function Write-ZtProgress {}
53+
54+
function Get-ZtTest {
55+
param([switch] $Current, $Tests)
56+
return [PSCustomObject]@{
57+
TestId = '99999'
58+
Title = 'Canary test'
59+
Pillar = 'Identity'
60+
SfiPillar = $null
61+
MinimumLicense = $null
62+
CompatibleLicense = $null
63+
}
64+
}
65+
}
66+
67+
It "formats Graph failures with safe troubleshooting details only" {
68+
$errorRecord = New-CanaryErrorRecord
69+
$test = [PSCustomObject]@{ TestID = '99999' }
70+
71+
$result = Format-ZtTestErrorDetail -Test $test -ErrorRecord $errorRecord
72+
73+
$result | Should -Match 'Graph request failed: GET https://graph.microsoft.com/beta/example'
74+
$result | Should -Match 'HTTP status: 401 Unauthorized'
75+
$result | Should -Match 'Graph error code: UnknownError'
76+
$result | Should -Match 'Request ID: 11111111-1111-1111-1111-111111111111'
77+
$result | Should -Match 'Client request ID: 22222222-2222-2222-2222-222222222222'
78+
$result | Should -Match 'HTTP Status Code: 401'
79+
$result | Should -Match 'GraphRequestFailed'
80+
$result | Should -Not -Match [regex]::Escape($script:canaryToken)
81+
$result | Should -Not -Match 'Authorization:|Cookie:|example response body|access_token='
82+
}
83+
84+
It "stores and logs only a sanitized error record for parallel failures" {
85+
$errorRecord = New-CanaryErrorRecord
86+
$test = [PSCustomObject]@{ TestID = '99999'; Title = 'Canary test' }
87+
$executionResult = [PSCustomObject]@{ Success = $true; Error = $null; DisplayName = 'Canary test' }
88+
89+
Mock Add-ZtTestResultDetail {
90+
param($TestId, $Title, $Status, $Result, $CustomStatus)
91+
$script:addedResult = $Result
92+
}
93+
94+
Write-ZtTestError -Test $test -Result $executionResult -ErrorRecord $errorRecord
95+
96+
$executionResult.Error.TargetObject | Should -BeNullOrEmpty
97+
$script:loggedErrorRecord.TargetObject | Should -BeNullOrEmpty
98+
$script:addedResult | Should -Not -Match [regex]::Escape($script:canaryToken)
99+
$script:addedResult | Should -Not -Match 'Authorization:'
100+
}
101+
102+
It "redacts credential headers at the TestResult persistence boundary" {
103+
$unsafeResult = @(
104+
'Authorization: Bearer example-canary-token'
105+
'Cookie: session=example-cookie'
106+
'X-Api-Key: example-api-key'
107+
'https://example.test/callback?access_token=example-access-token&client_secret=example-client-secret&sig=example-signature'
108+
) -join "`n"
109+
110+
Add-ZtTestResultDetail -TestId '99999' -Title 'Canary test' -Description 'Canary description' -Status $false -Result $unsafeResult -CustomStatus Error
111+
112+
$storedResult = $script:__ZtSession.TestResultDetail.Value['99999'].TestResult
113+
$storedResult | Should -Not -Match 'example-canary-token|example-cookie|example-api-key|example-access-token|example-client-secret|example-signature'
114+
$storedResult | Should -Match '<redacted>'
115+
}
116+
117+
It "writes only the safe error summary to optional test logs" {
118+
$errorRecord = New-CanaryErrorRecord
119+
$safeError = New-ZtSafeErrorRecord -ErrorRecord $errorRecord
120+
$logsPath = Join-Path $TestDrive 'logs'
121+
$test = [PSCustomObject]@{ TestID = '99999'; Title = 'Canary test' }
122+
$executionResult = [PSCustomObject]@{
123+
TestID = '99999'
124+
Test = $test
125+
Success = $false
126+
TimedOut = $false
127+
Duration = [TimeSpan]::Zero
128+
Start = Get-Date
129+
End = Get-Date
130+
Error = $safeError
131+
Messages = @()
132+
}
133+
134+
Write-ZtTestLog -Result $executionResult -LogsPath $logsPath
135+
136+
$logContent = Get-Content (Join-Path $logsPath '2-Tests/99999.md') -Raw
137+
$logContent | Should -Not -Match [regex]::Escape($script:canaryToken)
138+
$logContent | Should -Not -Match 'Authorization:'
139+
}
140+
}

‎src/powershell/private/core/Add-ZtTestResultDetail.ps1‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,8 @@ function Add-ZtTestResultDetail {
177177
$Result = $Result -replace "%TestResult%", $graphResultMarkdown
178178
}
179179

180+
$Result = Protect-ZtReportText -Text $Result
181+
180182
# Check if the docs team have provided a title for the test and use it if available
181183

182184
$docsTitle = $Title # Default to the title provided in the parameter

‎src/powershell/private/core/Protect-ZtReportText.ps1‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
function Protect-ZtReportText {
2+
<#
3+
.SYNOPSIS
4+
Removes common HTTP credentials from report content.
5+
6+
.DESCRIPTION
7+
Redacts credential-bearing HTTP headers and bearer/basic authorization
8+
values before text is persisted in assessment artifacts. This is a
9+
defense-in-depth control; callers should not serialize arbitrary request
10+
or response objects into report content.
11+
#>
12+
[CmdletBinding()]
13+
[OutputType([string])]
14+
param (
15+
[AllowNull()]
16+
[string]
17+
$Text
18+
)
19+
process {
20+
if ($null -eq $Text) {
21+
return $null
22+
}
23+
24+
$credentialHeaderPattern = '(?im)(\b(?:proxy-)?authorization|\bcookie|\bset-cookie|\bx-api-key|\bapi-key|\bocp-apim-subscription-key)\s*[:=]\s*[^\r\n,;}\]]+'
25+
$sanitizedText = [regex]::Replace($Text, $credentialHeaderPattern, '$1: <redacted>')
26+
$bearerPattern = '(?i)\bBearer\s+[A-Za-z0-9\-._~+/]+=*'
27+
$sanitizedText = [regex]::Replace($sanitizedText, $bearerPattern, 'Bearer <redacted>')
28+
$querySecretPattern = '(?i)(\b(?:access_token|client_secret|refresh_token|id_token|assertion|sig|token)\b\s*=\s*)[^&\s"''\r\n]+'
29+
30+
return [regex]::Replace($sanitizedText, $querySecretPattern, '$1<redacted>')
31+
}
32+
}

‎src/powershell/private/tests/Format-ZtTestErrorDetail.ps1‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
function Format-ZtTestErrorDetail {
2+
<#
3+
.SYNOPSIS
4+
Formats safe diagnostic details for a failed assessment test.
5+
6+
.DESCRIPTION
7+
Creates the existing failed-test markdown structure from allow-listed
8+
error properties. It intentionally does not format arbitrary ErrorRecord
9+
properties such as TargetObject, request headers, or exception data.
10+
#>
11+
[CmdletBinding()]
12+
[OutputType([string])]
13+
param (
14+
[Parameter(Mandatory)]
15+
$Test,
16+
17+
[Parameter(Mandatory)]
18+
[System.Management.Automation.ErrorRecord]
19+
$ErrorRecord
20+
)
21+
process {
22+
$safeError = New-ZtSafeErrorRecord -ErrorRecord $ErrorRecord
23+
$details = [System.Collections.Generic.List[string]]::new()
24+
$details.Add("Exception Type: $($ErrorRecord.Exception.GetType().FullName)")
25+
$details.Add("Fully Qualified Error ID: $($safeError.FullyQualifiedErrorId)")
26+
27+
$statusCode = Get-ZtHttpStatusCode -ErrorRecord $ErrorRecord
28+
if ($null -ne $statusCode) {
29+
$details.Add("HTTP Status Code: $statusCode")
30+
}
31+
32+
if ($ErrorRecord.InvocationInfo -and $ErrorRecord.InvocationInfo.ScriptName) {
33+
$details.Add("Location: $($ErrorRecord.InvocationInfo.ScriptName):$($ErrorRecord.InvocationInfo.ScriptLineNumber)")
34+
}
35+
36+
return @(
37+
'❌ Test {0} failed due to an unexpected error.' -f $Test.TestID
38+
' - **Error Message**: {0}.' -f $safeError.Exception.Message
39+
'```'
40+
($details -join [Environment]::NewLine)
41+
'```'
42+
) -join "`r`n"
43+
}
44+
}

‎src/powershell/private/tests/Get-ZtSafeErrorMessage.ps1‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
function Get-ZtSafeErrorMessage {
2+
<#
3+
.SYNOPSIS
4+
Builds a diagnostic summary that is safe to persist.
5+
6+
.DESCRIPTION
7+
Extracts a small allow-list of HTTP and Microsoft Graph diagnostic fields
8+
from an ErrorRecord. It never returns an unstructured exception message,
9+
request header dump, response body, or URL query string.
10+
#>
11+
[CmdletBinding()]
12+
[OutputType([string])]
13+
param (
14+
[Parameter(Mandatory)]
15+
[System.Management.Automation.ErrorRecord]
16+
$ErrorRecord
17+
)
18+
process {
19+
$diagnostics = [System.Collections.Generic.List[string]]::new()
20+
$errorText = @(
21+
$ErrorRecord.Exception.Message
22+
$ErrorRecord.ErrorDetails.Message
23+
) -join [Environment]::NewLine
24+
25+
$requestMatch = [regex]::Match($errorText, '(?i)\b(GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS)\s+(https?://[^\s"'']+)')
26+
if ($requestMatch.Success) {
27+
$method = $requestMatch.Groups[1].Value.ToUpperInvariant()
28+
$requestUri = $requestMatch.Groups[2].Value
29+
try {
30+
$uri = [System.Uri]$requestUri
31+
$requestUri = $uri.GetLeftPart([System.UriPartial]::Path)
32+
}
33+
catch {
34+
$requestUri = $requestUri -replace '\?[^\s"'']*$', ''
35+
}
36+
$diagnostics.Add("Graph request failed: $method $requestUri")
37+
}
38+
39+
$statusCode = Get-ZtHttpStatusCode -ErrorRecord $ErrorRecord
40+
if ($null -ne $statusCode) {
41+
$reasonMatch = [regex]::Match($errorText, "(?im)^HTTP/\S+\s+$statusCode\s+([^\r\n]+)")
42+
$reason = if ($reasonMatch.Success) { $reasonMatch.Groups[1].Value.Trim() } else { $null }
43+
$statusDescription = if ($reason) { "$statusCode $reason" } else { "$statusCode" }
44+
$diagnostics.Add("HTTP status: $statusDescription")
45+
}
46+
47+
$graphCodeMatch = [regex]::Match($errorText, '"code"\s*:\s*"([A-Za-z0-9_.-]+)"')
48+
if ($graphCodeMatch.Success) {
49+
$diagnostics.Add("Graph error code: $($graphCodeMatch.Groups[1].Value)")
50+
}
51+
52+
$requestIdMatch = [regex]::Match($errorText, '(?im)^\s*request-id\s*:\s*([0-9a-f-]{36})\s*$')
53+
if ($requestIdMatch.Success) {
54+
$diagnostics.Add("Request ID: $($requestIdMatch.Groups[1].Value)")
55+
}
56+
57+
$clientRequestIdMatch = [regex]::Match($errorText, '(?im)^\s*client-request-id\s*:\s*([0-9a-f-]{36})\s*$')
58+
if ($clientRequestIdMatch.Success) {
59+
$diagnostics.Add("Client request ID: $($clientRequestIdMatch.Groups[1].Value)")
60+
}
61+
62+
if ($diagnostics.Count -eq 0) {
63+
return 'An unexpected error occurred. See the error details for the exception type and error ID.'
64+
}
65+
66+
$diagnostics -join '; '
67+
}
68+
}

‎src/powershell/private/tests/Invoke-ZtTest.ps1‎

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -168,18 +168,13 @@
168168
}
169169
}
170170
catch {
171-
Write-PSFMessage -Level Warning -Message "Error executing test '{0}': {1}" -StringValues $Test.TestID, $_.Exception.Message -Target $Test -ErrorRecord $_
171+
$safeError = New-ZtSafeErrorRecord -ErrorRecord $_
172+
Write-PSFMessage -Level Warning -Message "Error executing test '{0}': {1}" -StringValues $Test.TestID, $safeError.Exception.Message -Target $Test -ErrorRecord $safeError
172173
$result.Success = $false
173-
$result.Error = $_
174-
$message = @(
175-
'❌ Test {0} failed due to an unexpected error.' -f $Test.TestID
176-
' - **Error Message**: {0}.' -f $_.Exception.Message
177-
'```'
178-
'{0}' -f ($_ | Get-Error | Out-String)
179-
'```'
180-
) -join "`r`n"
174+
$result.Error = $safeError
175+
$message = Format-ZtTestErrorDetail -Test $Test -ErrorRecord $_
181176
Add-ZtTestResultDetail -TestId $Test.TestID -Title $Test.Title -Status $false -Result $message -CustomStatus 'Error'
182-
Update-ZtProgressState -WorkerId $Test.TestID -WorkerName $result.DisplayName -WorkerStatus 'Error' -WorkerDetail "Error: $($_.Exception.Message)"
177+
Update-ZtProgressState -WorkerId $Test.TestID -WorkerName $result.DisplayName -WorkerStatus 'Error' -WorkerDetail "Error: $($safeError.Exception.Message)"
183178
}
184179
finally {
185180
$result.End = Get-Date

‎src/powershell/private/tests/New-ZtSafeErrorRecord.ps1‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
function New-ZtSafeErrorRecord {
2+
<#
3+
.SYNOPSIS
4+
Creates a safe copy of an error record for assessment diagnostics.
5+
6+
.DESCRIPTION
7+
Builds a new ErrorRecord without the original TargetObject, which can
8+
contain HTTP request or response objects with credentials. Its message is
9+
rebuilt from allow-listed diagnostic fields before use in reports and
10+
diagnostic artifacts.
11+
#>
12+
[CmdletBinding()]
13+
[OutputType([System.Management.Automation.ErrorRecord])]
14+
param (
15+
[Parameter(Mandatory)]
16+
[System.Management.Automation.ErrorRecord]
17+
$ErrorRecord
18+
)
19+
process {
20+
$message = Get-ZtSafeErrorMessage -ErrorRecord $ErrorRecord
21+
$exception = [System.Exception]::new($message)
22+
$errorId = $ErrorRecord.FullyQualifiedErrorId
23+
24+
return [System.Management.Automation.ErrorRecord]::new(
25+
$exception,
26+
$errorId,
27+
$ErrorRecord.CategoryInfo.Category,
28+
$null
29+
)
30+
}
31+
}

‎src/powershell/private/tests/Write-ZtTestError.ps1‎

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -33,17 +33,12 @@
3333
$ErrorRecord
3434
)
3535
process {
36-
Write-PSFMessage -Level Warning -Message "Error executing test '{0}': {1}" -StringValues $Test.TestID, $ErrorRecord.Exception.Message -Target $Test -ErrorRecord $ErrorRecord
36+
$safeError = New-ZtSafeErrorRecord -ErrorRecord $ErrorRecord
37+
Write-PSFMessage -Level Warning -Message "Error executing test '{0}': {1}" -StringValues $Test.TestID, $safeError.Exception.Message -Target $Test -ErrorRecord $safeError
3738
$Result.Success = $false
38-
$Result.Error = $ErrorRecord
39-
$message = @(
40-
'❌ Test {0} failed due to an unexpected error.' -f $Test.TestID
41-
' - **Error Message**: {0}.' -f $ErrorRecord.Exception.Message
42-
'```'
43-
'{0}' -f ($ErrorRecord | Get-Error | Out-String)
44-
'```'
45-
) -join "`r`n"
39+
$Result.Error = $safeError
40+
$message = Format-ZtTestErrorDetail -Test $Test -ErrorRecord $ErrorRecord
4641
Add-ZtTestResultDetail -TestId $Test.TestID -Title $Test.Title -Status $false -Result $message -CustomStatus 'Error'
47-
Update-ZtProgressState -WorkerId $Test.TestID -WorkerName $Result.DisplayName -WorkerStatus 'Error' -WorkerDetail "Error: $($ErrorRecord.Exception.Message)"
42+
Update-ZtProgressState -WorkerId $Test.TestID -WorkerName $Result.DisplayName -WorkerStatus 'Error' -WorkerDetail "Error: $($safeError.Exception.Message)"
4843
}
4944
}

0 commit comments

Comments
 (0)