You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: .github/workflows/objective-impact-report.md
+26-34Lines changed: 26 additions & 34 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,21 +19,20 @@ safe-outputs:
19
19
20
20
## Goal
21
21
22
-
Test whether Impact Efficiency is a more meaningful signal than accepted outcome counts alone.
22
+
POC: Test whether Impact Efficiency is a more meaningful signal than accepted outcome counts alone.
23
+
24
+
Focus only on **pull request outcomes** and **safe output outcomes** (issues created or closed via the safe-output mechanism). Other outcome types are excluded from this POC because their acceptance criteria are not yet well-defined and most remain pending.
23
25
24
26
Use this model:
25
27
26
28
```text
27
-
Outcome = recorded work item produced by a GitHub Agentic Workflow run
28
-
Objective = issue/epic/work item linked to the outcome
29
+
Outcome = a PR or safe-output issue produced by a GitHub Agentic Workflow run
29
30
Objective Value = numeric value from the repository objective-mapping configuration applied to traced root labels
30
31
Outcome Indicator = 1 for accepted/delivered outcomes, 0 otherwise
31
32
Outcome Value = Outcome Indicator × Objective Value
32
33
Impact Efficiency = Σ Outcome Value / AI Credits
33
34
```
34
35
35
-
Treat an outcome as one recorded result item produced by a GitHub Agentic Workflow run (for example, a PR change, completed fix, or report action), which may later be accepted or not accepted.
36
-
Use workflow run outputs/artifacts and linked GitHub objects (issues, PRs, comments, discussions) as the outcome source of truth.
37
36
Treat AI Credits as total model-credit cost consumed by the workflow runs that produced the analyzed outcomes.
38
37
When available, use deterministic precomputed run data that already includes each run's `aic` field.
39
38
Prefer existing gh-aw outputs that already surface `aic`, such as pre-downloaded `gh aw logs --json` data or audit/log artifacts derived from the same run summaries.
@@ -58,7 +57,12 @@ If a run's `aic` field is missing or null, treat it as `0` and count it as missi
58
57
59
58
## Scope
60
59
61
-
Analyze workflow outcomes and linked objectives from the last 180 days.
60
+
Analyze only the following outcome types from the last 180 days:
61
+
62
+
-**Pull request outcomes**: PRs created by GitHub Agentic Workflow runs. Accepted = merged. Rejected = closed without merge. Skip open (pending) PRs.
63
+
-**Safe output outcomes**: issues created or closed by workflow runs via the safe-output mechanism. Accepted = issue successfully created/closed. Skip any with unresolved state.
64
+
65
+
Exclude all other outcome types (direct issue outcomes, comments, discussions, etc.). These are omitted because their acceptance criteria are incomplete and most are left pending, which would distort the metric.
Do not invent fallback scoring rules such as milestone bonuses, project bonuses, or priority-to-points heuristics when the mapping file is present.
83
87
84
-
```text
85
-
Examples of mapped labels in this repository include campaign, security, observability, testing, automation, and other configured objective labels.
86
-
```
87
-
88
88
If a traced root object has no labels that exist in the mapping, mark the outcome as `unmapped`.
89
89
90
90
## Outcome association rules
91
91
92
-
For each workflow outcome, follow the implemented root-tracing behavior:
92
+
For each in-scope outcome, follow the implemented root-tracing behavior:
93
93
94
94
1. For pull-request outcomes, trace the PR to its linked closing issue and use that root issue's labels.
95
-
2. If PR root tracing fails, or for direct issue outcomes, use labels on the issue itself.
96
-
3. Record the traced root URL when one is found so the report preserves an audit trail.
97
-
4. If no mapped objective labels can be found, mark the outcome as `unmapped`, exclude it from `Σ Outcome Value`, and report it separately.
98
-
99
-
Prefer precomputed outcome evaluation data when available. Do not re-derive a different mapping model inside the report.
95
+
2. If PR root tracing fails, use labels on the PR itself.
96
+
3. For safe-output issue outcomes, use labels on the safe-output issue itself.
97
+
4. Record the traced root URL when one is found so the report preserves an audit trail.
98
+
5. If no mapped objective labels can be found, mark the outcome as `unmapped`, exclude it from `Σ Outcome Value`, and report it separately.
100
99
101
100
## Computation
102
101
103
-
For each outcome:
102
+
For each in-scope outcome:
104
103
105
104
```text
106
-
Outcome Indicator = 1 for accepted/delivered outcome, 0 for rejected, abandoned, or incomplete outcome
105
+
Outcome Indicator:
106
+
PR outcome: 1 if merged, 0 if closed without merge (open PRs excluded)
107
+
Safe output outcome: 1 if successfully created/closed, 0 otherwise
107
108
Outcome Value = Outcome Indicator × Objective Value
108
109
```
109
110
110
-
Treat pending-review outcomes as `Outcome Indicator = 0` until explicitly accepted.
111
-
112
-
Accepted/delivered outcome means the intended result was accepted in GitHub state (for example: merged PR, closed issue with completion signal, or explicit accepted status in the workflow outcome record).
0 commit comments