Skip to content

Enable human-gated release-readiness PR creation - #37693

Open
kubaflo wants to merge 6 commits into
dotnet:mainfrom
kubaflo:feature/release-readiness-backport-execution
Open

Enable human-gated release-readiness PR creation#37693
kubaflo wants to merge 6 commits into
dotnet:mainfrom
kubaflo:feature/release-readiness-backport-execution

Conversation

@kubaflo

@kubaflo kubaflo commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Summary

  • keep the deterministic release-readiness scripts report-only
  • allow the interactive agent, after an explicit request, to prepare any scoped release repository change that can be reviewed as a PR—not only backports
  • require agent-created PRs to use a release-agent/* fork branch and a durable PR-body marker; Arcade backport/pr-* PRs are detected separately
  • add a fail-closed Release Agent Human Approval workflow requiring two approvals on the current head SHA from distinct non-bot MAUI maintainers with write access, excluding the PR author
  • retain hard prohibitions on direct upstream pushes, self-approval, merge/auto-merge, branch cuts, tags, releases, and build or deployment triggers

Approval enforcement

The workflow runs in a trusted context without checking out or executing pull-request code. It fully paginates review history, collapses each reviewer to their latest decision-bearing review, rejects stale-head approvals, filters GitHub Apps, [bot] users and known PAT-based automation accounts, and verifies effective repository permission before counting an approval.

After this PR merges, repository administrators must configure Release Agent Human Approval / Require human approval as a required status check on each protected target branch or ruleset. The workflow cannot make itself required through a repository-content PR, and it supplements rather than replaces the normal two-review policy.

Validation

  • pwsh -NoProfile -File .github/skills/release-readiness/tests/Test-ReleaseAgentHumanApproval.ps1 — 14 passed
  • pwsh -NoProfile -File .github/skills/release-readiness/tests/Test-ReleaseReadiness.ps1 -SkipE2E — 2,153 passed
  • actionlint .github/workflows/release-agent-human-approval.yml
  • git diff --check

Keep readiness read-only by default while allowing explicitly authorized Arcade commands and fork-based manual backport PRs. Add contract tests for authorization, duplicate detection, main ancestry, and direct-release mutation safeguards.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1e7f77e3-4862-4b92-a662-d12b5918bb59
Copilot AI lite review requested due to automatic review settings August 20, 2026 22:49
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 20, 2026 22:49 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 37693

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 37693"

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 20, 2026 22:49 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Skill Validation Results

@kubaflo — new skill validation results are available based on this last commit: decac9d.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

Overall Failed Static Failed LLM Needs attention Skills 0 Agents 0

Skill Validation Resultsdecac9d · Enable human-gated release-readiness PR creation · 2026-08-27T02:15:13Z

❌ Static Checks Failed

Skills: 0 | Eval specs linted:

⏭️ LLM Evaluation: Skipped

No changed skills with eval specs found.

🔍 Full results and investigation steps

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 20, 2026 22:50 — with GitHub Actions Inactive
@kubaflo kubaflo added the area-ai-agents Copilot CLI agents, agent skills, AI-assisted development label Aug 20, 2026
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 20, 2026 22:52 — with GitHub Actions Inactive
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 20, 2026 22:53 — with GitHub Actions Inactive
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool August 20, 2026 22:54 — with GitHub Actions Inactive

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens the separation between the deterministic release-readiness PowerShell engine (report-only) and the interactive release-readiness-agent by explicitly documenting a constrained “action mode” for SR backports that only activates after an explicit user request, and by adding contract tests to enforce that boundary in documentation.

Changes:

  • Document an explicit, tightly-scoped SR backport execution path for the interactive release-readiness-agent, while keeping deterministic scripts report-only.
  • Expand methodology guidance for automated vs manual (fork-based) backport paths, including “non-main selected candidate commit” exception rules under explicit release-captain authorization.
  • Add unit/contract tests that assert the presence of required safety gates and prohibitions across the agent contract, skill docs, and methodology reference.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
.github/skills/release-readiness/tests/Test-ReleaseReadiness.ps1 Adds contract assertions that documentation continues to encode the intended read-only default and constrained action gates.
.github/skills/release-readiness/SKILL.md Clarifies deterministic vs interactive responsibilities; explicitly allows authorized SR backport follow-up without introducing write behavior into the skill scripts.
.github/skills/release-readiness/references/methodology.md Documents the execution boundary, required pre-write checks, and the fork-only manual PR workflow (including non-main exception constraints).
.github/copilot-instructions.md Updates repository-level agent/skill guidance so delegation and “explicit authorization” behavior is consistently documented.
.github/agents/release-readiness-agent.agent.md Defines the “read-only by default; explicit backports only” contract, including explicit gates and hard prohibitions on release-ref mutation, merges, tags, builds, and scope creep.

Generalize explicit release actions to fork-based PRs and add a fail-closed approval gate for the current PR head.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 1e7f77e3-4862-4b92-a662-d12b5918bb59
Copilot AI review requested due to automatic review settings August 20, 2026 23:04
@kubaflo kubaflo changed the title Enable explicitly authorized release-readiness backports Enable human-gated release-readiness PR creation Aug 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/release-agent-human-approval.yml:108

  • If the PowerShell policy throws before emitting JSON (e.g., invalid/missing input, download failure), $result will be empty/non-JSON. With set -e, the subsequent jq calls will fail and the step will exit due to jq instead of the policy’s failure, which can mask the real error and prevent a useful step summary. Consider guarding the jq parsing (e.g., if jq -e . "$result" >/dev/null 2>&1; then ... fi) and preserving exit "$status" as the only failure signal when JSON isn’t available.
          result="$RUNNER_TEMP/human-approval.json"
          set +e
          pwsh -NoProfile -File "$POLICY_PATH" \
            -ReviewsJsonPath "$REVIEWS_PATH" \
            -ReviewerPermissionsJsonPath "$REVIEWER_PERMISSIONS_PATH" \
            -PullRequestAuthor "$PULL_REQUEST_AUTHOR" \
            -PullRequestHeadSha "$PULL_REQUEST_HEAD_SHA" \
            -RequiredApprovals 2 > "$result"
          status=$?
          set -e

          cat "$result"
          {
            echo "### Release-agent human approval"
            echo
            echo "- Required approvals: $(jq -r '.RequiredApprovals' "$result")"
            echo "- Qualified approvers: $(jq -r 'if (.Approvers | length) == 0 then "_none_" else (.Approvers | join(", ")) end' "$result")"
          } >> "$GITHUB_STEP_SUMMARY"

@PureWeen PureWeen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review

Policy Enforcement — The job applicability at .github/workflows/release-agent-human-approval.yml:20-23 is controlled only by the PR head-name convention or a literal, author-editable body marker. A release-agent action that creates a nonconforming branch, or a marker-only PR whose author later removes the marker, causes this required job to be skipped; skipped required jobs satisfy branch protection rather than enforcing the two-approval policy. Use a trusted, non-author-controlled provenance signal, or an always-running gate that fails closed when an intended release-agent PR is not positively identified. Flagged by: 2/3 reviewers.

⚠️ Regression Prevention.github/workflows/release-readiness.yml:40 adds approval-workflow changes to the PR-validation paths, but the same validation runs new assertions over .github/agents/release-readiness-agent.agent.md without including that agent contract in paths. A PR that weakens only the agent action boundary skips the workflow entirely, leaving those guards unexecuted. Add the agent file (or .github/agents/**) to the trigger. Flagged by: 3/3 reviewers.

⚠️ Error Handling — The permission lookup at .github/workflows/release-agent-human-approval.yml:57-59 runs under set -e; a deleted/renamed reviewer or transient API failure aborts the whole required gate even if two other current-head approvals qualify. The PowerShell policy already treats missing permission evidence as untrusted, so make each failed lookup record none and continue rather than turning an irrelevant reviewer into a merge-blocking outage. Flagged by: 3/3 reviewers.

Review method: 3 independent reviewers with adversarial consensus. The existing malformed-policy-output observation was not duplicated. The added approval-policy tests cover several decision-ordering cases, but the workflow trigger does not currently cover standalone agent-contract changes.

@PureWeen PureWeen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review follow-up: independent reviewer assessment confirms that the three actionable issues from review 4988665802 remain. No additional analysis was performed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d00747b7-96f3-4e7a-8dfb-e3a48db04b2d
Copilot AI review requested due to automatic review settings August 21, 2026 13:04
@kubaflo

kubaflo commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

@PureWeen addressed the latest feedback in 8c6e39c: the required job now always reports and preserves applicability through trusted github-actions[bot] label history; agent-contract-only changes trigger validation; failed permission lookups remain untrusted without aborting other approvals; and malformed policy output no longer masks the policy exit status. The focused approval tests pass 14/14, the release-readiness suite passes 2291/2291, and the workflows pass actionlint. This is ready for re-review — thanks!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

@PureWeen PureWeen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review

No actionable issues found at 8c6e39c8ed773772ec364944eba10dd4f302847e. Three independent reviewer assessments examined the trusted pull_request_target boundary, durable applicability tracking, current-head approval policy, error paths, documentation contract, and targeted regression coverage.

The only potential operational concern was independently re-evaluated and discarded: the bounded, sequential permission lookups intentionally fail closed and do not present a concrete PR-introduced regression.

Test coverage assessment: the focused approval-policy tests cover current-head binding, latest review decisions, author/bot exclusion, permissions, and malformed data; release-readiness validation now runs when the agent contract or approval workflow changes.

Method: 3 independent reviewers with adversarial consensus. This independent reviewer assessment posts a COMMENT-only verdict.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d00747b7-96f3-4e7a-8dfb-e3a48db04b2d
Copilot AI review requested due to automatic review settings August 27, 2026 00:16
@kubaflo

kubaflo commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Merged current main into this branch in d9ddcac. The release-readiness overlap retains main’s RC/all-lane support alongside this PR’s human-approval guard. The focused approval and release-readiness suites passed (14 + 2,349 assertions), the new approval workflow is actionlint-clean, and GitHub reports the PR MERGEABLE.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Comment thread .github/workflows/release-agent-human-approval.yml
Comment thread .github/skills/release-readiness/references/methodology.md Outdated
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d00747b7-96f3-4e7a-8dfb-e3a48db04b2d
Copilot AI review requested due to automatic review settings August 27, 2026 00:51
@kubaflo

kubaflo commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

@copilot-pull-request-reviewer addressed the latest feedback in e2ea88d21e (fail-closed label provisioning and remote-neutral backport instructions, covered by 16 focused checks). This is ready for re-review — thanks!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/skills/release-readiness/tests/Test-ReleaseAgentHumanApproval.ps1:99

  • This test uses a lowercased mauibot login, which would not catch a case-sensitivity bug in the known-bot deny-list. Use a mixed-case login to ensure the policy treats known automation accounts case-insensitively.
$botApproval = Get-TestResult @(
    (New-Review 4 'mauibot' 'APPROVED' '2026-01-01T00:03:00Z'),
    $aliceApproval
)
Assert-Equal 'PAT-based bot account cannot satisfy the gate' $false $botApproval.Approved

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d00747b7-96f3-4e7a-8dfb-e3a48db04b2d
Copilot AI review requested due to automatic review settings August 27, 2026 02:14
@kubaflo

kubaflo commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

@copilot-pull-request-reviewer addressed the case-insensitive automation-account feedback in decac9d438, including mixed-case regression coverage. The focused approval-policy suite passes 16/16. This is ready for re-review — thanks!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-ai-agents Copilot CLI agents, agent skills, AI-assisted development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants