feat: pre-flight PR deduplication protocol to cut Copilot retry waste (#28843)

Author: Copilot

.github/agents/developer.instructions.md

@@ -22,6 +22,7 @@ This document consolidates development guidelines, architectural patterns, and i
 - [Hierarchical Agent Management](#hierarchical-agent-management)
 - [Release Management](#release-management)
 - [Scope Hints for Complex Workflows](#scope-hints-for-complex-workflows)
+- [PR Deduplication Protocol](#pr-deduplication-protocol)
 - [Quick Reference](#quick-reference)
 
 ---
@@ -700,7 +701,63 @@ Before submitting a complex workflow request, confirm you have specified:
 
 ---
 
-## Quick Reference
+## PR Deduplication Protocol
+
+Repeated closed PR attempts on the same topic waste CI resources and agent context. Follow this protocol every time you are about to create a pull request.
+
+### Pre-flight Duplicate PR Check
+
+Before opening a PR, search for existing closed PRs with a similar topic using the GitHub MCP `search_pull_requests` tool:
+
+1. Extract 2–4 keywords from the feature/fix title.
+2. Run a search such as:
+   - `is:pr is:closed head:copilot/ <keywords>`
+   - `is:pr is:closed <keywords>`
+3. If **no** closed PR is found, proceed normally.
+4. If **one or more** closed PRs are found, move to the [Prior Failure Analysis](#prior-failure-analysis) step before writing any code.
+
+### Prior Failure Analysis
+
+When a closed PR exists on the same topic, perform this analysis at the start of the session — before any code exploration or implementation:
+
+1. Read the closed PR description, review comments, and timeline.
+2. Identify the **root cause of closure**:
+   - Reviewer requested changes → list them explicitly
+   - CI/test failures → identify failing checks and root cause
+   - Scope mismatch → clarify what was actually requested
+   - Duplicate of another fix → link to that fix
+3. Verify that the root cause will be addressed in the new implementation.
+4. Include a "## Prior Attempts" section in the new PR description that summarizes:
+   - Link(s) to prior closed PR(s)
+   - Why each was closed
+   - What is different this time
+
+**Example PR description section:**
+
+```markdown
+## Prior Attempts
+
+- #1234 (closed): CI failed on `TestFoo` due to missing nil check — fixed in this PR
+- #1189 (closed): Reviewer requested scope reduction — this PR limits change to X only
+```
+
+### Retry Limit Circuit Breaker
+
+If **two or more** closed PRs already exist on the same topic:
+
+1. **Do not open a third PR** without explicit human review.
+2. Post a comment on the originating issue that:
+   - Lists all prior closed PRs and their close reasons
+   - Explains what changed (if anything) in the new approach
+   - Requests explicit maintainer approval to proceed
+3. Label the issue `copilot-retry-blocked` to signal that human review is required.
+4. Wait for a maintainer to remove the label or leave an approving comment before creating the new PR.
+
+**Rationale:** Two consecutive failed PR attempts indicate a systemic problem (unclear requirements, missing context, fundamental design issue) that code changes alone cannot resolve.
+
+---
+
+
 
 ### File Locations
 
@@ -792,4 +849,4 @@ For detailed specifications, see individual files in `scratchpad/`:
 
 ---
 
-**Last Updated**: 2026-01-31
+**Last Updated**: 2026-04-28

.github/workflows/copilot-opt.lock.yml

@@ -34,6 +34,7 @@ safe-outputs:
     max: 3
     labels: [copilot-opt, optimization, cookie]
     title-prefix: "[copilot-opt] "
+    close-older-issues: true
 imports:
   - shared/jqschema.md
   - shared/copilot-session-data-fetch.md
@@ -59,7 +60,7 @@ Pre-fetched data is available from shared imports:
 
 - `/tmp/gh-aw/session-data/sessions-list.json`
 - `/tmp/gh-aw/session-data/logs/` (conversation logs and/or fallback logs)
-- `/tmp/gh-aw/pr-data/copilot-prs.json` (optional cross-analysis source)
+- `/tmp/gh-aw/pr-data/copilot-prs.json` (cross-analysis source — always present)
 
 These paths are populated by imported setup components:
 - `shared/copilot-session-data-fetch.md` writes the session files under `/tmp/gh-aw/session-data/`
@@ -77,12 +78,13 @@ These paths are populated by imported setup components:
    - large initial instruction/context payload
    - inefficient orchestration/model-loading patterns
    - prompt drift / instruction adherence degradation
-4. Optionally correlate findings with Copilot PR patterns from `/tmp/gh-aw/pr-data/copilot-prs.json` when useful.
-5. Generate **exactly three** recommendations:
+4. **Always** correlate findings with Copilot PR patterns from `/tmp/gh-aw/pr-data/copilot-prs.json`.
+5. **Always** perform duplicate PR pattern detection (see Phase 3) and surface retry-blocked topics.
+6. Generate **exactly three** recommendations:
    - each recommendation must target a distinct root cause
    - each recommendation must be concrete and actionable
    - each recommendation must include expected impact
-6. Create **exactly three GitHub issues** (one per recommendation).
+7. Create **exactly three GitHub issues** (one per recommendation).
 
 If data is incomplete, proceed with available evidence and clearly state data quality limitations.
 
@@ -127,15 +129,53 @@ For each session summary:
 
 Aggregate across all sessions to identify recurring systemic patterns.
 
-## Phase 3 — Optional PR Cross-Analysis
+## Phase 3 — PR Cross-Analysis and Duplicate Pattern Detection
 
-If `/tmp/gh-aw/pr-data/copilot-prs.json` is present and non-empty:
+This phase is **mandatory**. `/tmp/gh-aw/pr-data/copilot-prs.json` is always present from the imported `shared/copilot-pr-data-fetch.md` step.
+
+### 3a — General PR Failure Signals
 
 1. Extract recurring failure/friction signals from recent Copilot PRs.
-2. Correlate with session-derived patterns.
+2. Correlate with session-derived patterns from Phase 2.
 3. Increase priority for overlapping problem areas.
 
-If PR data is unavailable, continue without this phase and note that in evidence.
+### 3b — Duplicate PR Pattern Detection
+
+Identify topics where Copilot PRs were closed without merging and then re-attempted. This is the costliest waste pattern because each retry consumes a full agent session.
+
+**Detection procedure:**
+
+```bash
+# Find closed (not merged) PRs grouped by normalized title
+jq '[.[] | select(.state == "CLOSED" and .mergedAt == null)]
+    | group_by(.title)
+    | map({title: .[0].title, count: length, prs: [.[] | {number, url, closedAt}]})
+    | map(select(.count >= 2))
+    | sort_by(-.count)' /tmp/gh-aw/pr-data/copilot-prs.json
+```
+
+For each topic with **two or more** closed-without-merge PRs (retry-blocked topics):
+
+1. Record the PR numbers, titles, and close dates.
+2. Use the GitHub MCP `get_pull_request` or `search_pull_requests` tool to read the most recent closed PR and identify the close reason (review comments, CI failures, or reviewer request).
+3. Classify the close reason into one of:
+   - `ci-failure` — tests or lint failed
+   - `reviewer-rejected` — maintainer closed without merging and left a reason
+   - `scope-mismatch` — implementation did not match what was requested
+   - `duplicate` — a separate fix was merged that covers the same change
+   - `unknown` — no clear close reason found
+4. Build a **retry-blocked topics table**:
+
+   | Topic (PR title keywords) | Closed PRs | Close reason | Retry count |
+   |---------------------------|------------|--------------|-------------|
+   | …                         | #N, #M     | ci-failure   | 2           |
+
+### 3c — Retry Count Threshold
+
+- Topics with **exactly two** closed PRs: flag as **high-risk retry**. Include in recommendations if the close reason is actionable.
+- Topics with **three or more** closed PRs: flag as **retry-blocked**. These must produce a recommendation that explicitly calls for human review before any further agent attempt.
+
+If PR data is unexpectedly unavailable (file missing or empty), skip Phase 3 and note that in all three issue bodies.
 
 ## Phase 4 — Recommendation Selection
 
@@ -145,6 +185,7 @@ Selection rules:
 
 - cover distinct root causes (no overlap)
 - prioritize high-frequency and high-severity patterns
+- **retry-blocked topics (≥2 closed PRs) are automatically elevated to high priority** — if any exist, at least one recommendation must address them unless all three slots are taken by higher-impact findings from Phase 2
 - include evidence (counts, rates, or representative examples)
 - include expected impact and a concrete change proposal
 
@@ -155,6 +196,7 @@ Possible recommendation domains:
 - tool payload/latency optimization
 - earlier/stronger validation strategy
 - prompt design corrections to reduce drift
+- **duplicate-PR / retry waste reduction** (use when Phase 3b finds retry-blocked topics)
 
 ## Phase 5 — Issue Creation (Exactly Three)
 
@@ -192,6 +234,28 @@ Use this template:
 - Data quality caveats (if any)
 ```
 
+### Retry-Blocked Topic Addendum
+
+When an issue covers a retry-blocked topic (from Phase 3b), **append** the following section to the body:
+
+```markdown
+### Prior Failed Attempts
+
+The following Copilot PRs on this topic were closed without merging before this issue was created:
+
+| PR | Closed (YYYY-MM-DD) | Close reason |
+|----|---------------------|--------------|
+| #N | YYYY-MM-DD | [reason] |
+| #M | YYYY-MM-DD | [reason] |
+
+**Retry count: [N] — human review required before a new implementation attempt.**
+Any agent attempting to implement this recommendation MUST read this section and the linked PRs, address all close reasons, and post a plan comment on this issue before opening a new PR.
+```
+
+### Labels for Retry-Blocked Issues
+
+When creating an issue that covers a retry-blocked topic, add `copilot-retry-blocked` to the labels list in the `create_issue` safe-output call alongside the standard labels.
+
 ## Items That Should Not Be Addressed
 
 The following items are out of scope because they are not actionable by repository users:
@@ -215,8 +279,12 @@ Before creating issues, verify:
 - [ ] last-14-day filtering was applied
 - [ ] `events.jsonl` parsing was attempted across all in-scope sessions
 - [ ] tool latency/payload, validation timing, context size, orchestration, and prompt drift were analyzed
+- [ ] Phase 3 PR cross-analysis was performed (not skipped)
+- [ ] duplicate PR pattern detection was run and retry-blocked topics table was built
+- [ ] retry-blocked topics (≥2 closed PRs) are reflected in at least one recommendation when present
 - [ ] exactly three recommendations selected
 - [ ] each recommendation has evidence + proposed change + expected impact
+- [ ] retry-blocked issues include the "Prior Failed Attempts" addendum and `copilot-retry-blocked` label
 - [ ] exactly three issue outputs will be created
 
 ## Usage