Impact Efficiency Report - 2026-06-16

[github-actions]

Impact Efficiency Report — POC

Analysis date: 2026-06-16 · Target window: 180 days (2025-12-17 → 2026-06-16)
AIC source: Embedded in safe-output issue bodies (deterministic; PR-creating run AIC unavailable — see Data Quality)

Note

POC scope. This report validates whether Impact Efficiency is a more meaningful signal than raw accepted-outcome counts. Data coverage is partial: PRs span a 27-day sample (May 20 – Jun 16), safe-output issues a 5-day sample (Jun 10–15). Extrapolation across the full 180-day window would require additional pagination.

---

Summary

Metric	PRs	Safe-output Issues	Combined
Analyzed outcomes	763	500	1,263
Merged / created (accepted)	566	500	1,066
Closed / rejected	197	0	197
Open / excluded	—	—	—
Mapped outcomes	21 sampled (14 mapped)	209 / 500	223 mapped
Unmapped outcomes	7 sampled + ~742 unscored	291 / 500	~1,040
Accepted outcome count	566	500	1,066
Total outcome value	465 (21 sampled)	11,365	11,830
AI Credits (AIC)	unavailable	66,087.9 (319 / 500 with AIC)	66,087.9
Impact Efficiency	—	—	0.1790 value / AIC

AI Credits = 66,087.9 from safe-output issue-creating run costs only. PR-creating run AIC is not embedded in PR bodies and was not available from precomputed data. Impact Efficiency is therefore a lower bound (numerator complete for analyzed outcomes; denominator incomplete — excludes PR-creating costs).

---

Top outcomes by outcome value

Outcome	Type	Root / Associated Objective	Objective Value	Outcome Value
#39189 [static-analysis] Report - 2026-06-14	Safe-output issue	automation, security	85	85
#39025 [static-analysis] Report - 2026-06-13	Safe-output issue	automation, security	85	85
#38803 [Safe Outputs Conformance] SEC-004	Safe-output issue	security, safe-outputs	85	85
#38802 [Safe Outputs Conformance] SEC-005	Safe-output issue	security, safe-outputs	85	85
#38795 [static-analysis] Report - 2026-06-12	Safe-output issue	automation, security	85	85
#38525 [static-analysis] Report - 2026-06-11	Safe-output issue	automation, security	85	85
#39038 Failure Investigator Issue Group	Safe-output issue	bug, reliability, observability	70	70
#39037 Failure Investigator pre-fetch issue	Safe-output issue	bug, reliability, observability	70	70
#38992 Outcome Health Report — Jun 13	Safe-output issue	automation, observability	70	70
#38970 Daily Reliability Review - 2026-06-12	Safe-output issue	reliability, observability	70	70
#38835 Daily AIC Consumption Report - 2026-06-12	Safe-output issue	observability, telemetry	70	70
#38804 Daily Workflow Optimization - 2026-06-12	Safe-output issue	optimization, observability	70	70
PR #39378 [spec-enforcer] Enforce specifications	PR (merged)	automation, testing	65	65
PR #39329 [spec-review] Safe Outputs conformance	PR (merged)	automation, safe-outputs	55	55
PR #38185 [linter-miner] add execcommandwithoutcontext	PR (merged)	root #38143: improvement	45	45

---

Unmapped outcomes

Outcome	Type	Reason objective was not mapped
~291 safe-output issues	Safe-output issue	Dominant label agentic-workflows (261 occurrences) not in objective-mapping.json; also cascade-suspected (66), contribution-report (6), lgtm (6), triage-report (4)
~545 PRs (unscored)	PR	Labels not fetched (budget constraint in POC); likely a mix of unmapped and mappable labels
PR #39484, #39437, #39266, #39156	PR	No labels on PR; no linked closing issue found
PR #39439	PR	Labels are PR-management metadata only (pr-risk:low, pr-type:bug, pr-agent:copilot-swe-agent) — not in objective mapping
PR #38911	PR	Labels smoke, smoke-claude not in objective mapping
PR #39367, #39357, #39318	PR	Labels glossary, architecture, diagram, blog not in objective mapping
PR #36371	PR	Closed without merge (rejected); root issue maps to improvement=45 but Outcome Indicator = 0

Key gap: agentic-workflows is the single largest label (284/500 issues) but has no entry in .github/objective-mapping.json. Adding a value for this label (suggested ≥ 45) would more than double the mapped count and materially increase Total Outcome Value.

---

Interpretation

Accepted outcome count: 1,066

At face value, 1,066 accepted outcomes (566 merged PRs + 500 created/closed issues) looks like strong throughput. However, this count treats all outcomes as equal — a daily glossary update PR counts the same as a security-analysis issue or a safe-outputs conformance fix.

Impact Efficiency: 0.1790 value / AIC

The efficiency signal surfaces meaningful differences. The 6 static-analysis reports and 2 Safe Outputs Conformance issues (all at value 85) contribute 510 value to Total Outcome Value from 6 issue bodies. In contrast, 261 issues labeled only agentic-workflows contribute zero value despite being accepted outcomes — they represent real cost but zero mapped value. This category of outcomes looks identical in a count-based view but invisible in Impact Efficiency.

Conclusion: Impact Efficiency is the more meaningful signal. It forces the question: what does an accepted outcome actually achieve, relative to its cost? Accepted-count alone rewards volume; Impact Efficiency rewards alignment with declared objectives. For this repo, the metric reveals that:

1. Security and reliability outcomes (value ≥ 70) are disproportionately high-value per accepted count
2. A large fraction of safe-output issues (agentic-workflows-only labeled) are unmapped and contribute zero value despite accounting for significant AIC cost
3. PR outcomes from Copilot SWE agent (77% of agentic PRs) are nearly all unmapped, which is the primary gap in PR-side objective coverage

---

Data quality

PR root tracing and linked-issue coverage

• Of 763 agentic PRs analyzed, only 2 contained explicit Closes #N references to a linked issue. Root tracing coverage is <1%.
• Without linked issues, PR labels are the fallback. Only github-actions[bot] PRs (~175/763, 23%) consistently carry objective-relevant labels (testing, automation, safe-outputs). Copilot SWE-agent PRs (588/763, 77%) carry either no labels or PR-management labels (pr-risk, pr-type, pr-agent) not present in objective-mapping.json.
• Recommendation: Add labels to Copilot-generated PRs at creation time (e.g., from the issue they address), or populate Closes #N links systematically to enable root tracing.

Safe-output issue label mapping coverage

• agentic-workflows (284/500 issues, 57%) is the dominant label but has no entry in the mapping. This alone accounts for the majority of unmapped safe-output outcomes.
• Other unmapped labels: cascade-suspected, contribution-report, lgtm, triage-report, cookie, deep-report.
• Recommendation: Add agentic-workflows (suggested value ≥ 45) and deep-report (suggested ~40) to objective-mapping.json to close the largest unmapped gap.

AI Credits availability

• Source used: Deterministic values embedded in safe-output issue bodies (· N.N AIC pattern in workflow footers).
• Coverage: 319 / 500 issues (64%) had embedded AIC. The remaining 181 are treated as missing-cost data and contribute 0 to the denominator.
• PR-creating run AIC: Not available. The AIC values embedded in PR bodies (from PR Sous Chef, PR Description Updater) are for secondary workflow runs, not the creating agent session. No precomputed gh aw logs --json data with per-run aic fields was found in the repository.
• Impact on metric: Impact Efficiency = 0.1790 is a lower bound — the true denominator is higher once PR-creating costs are included, so the true IE is lower. The numerator (outcome value) is also a lower bound due to unmapped labels and unscored PRs.
• Recommendation: Expose per-run aic in a persistent artifact (e.g., gh aw logs --json export) so the full denominator can be computed.

---

Generated by Impact Efficiency Report workflow · github/gh-aw · 2026-06-16

Generated by 📊 Impact Efficiency Report · 1.1K AIC · ⌖ 26.1 AIC · ⊞ 29K · ◷