Skip to content

Commit 209fa60

Browse files
mnkiefermnkiefer
authored
docs: Add outcome span attributes and outcomes reference (#34627)
1 parent 1d20b06 commit 209fa60

3 files changed

Lines changed: 227 additions & 51 deletions

File tree

‎docs/src/content/docs/reference/open-telemetry.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,24 @@ and is emitted as `SPAN_KIND_CLIENT`.
165165
| `gen_ai.usage.cache_creation.input_tokens` | Cache-creation tokens written |
166166
| `gen_ai.response.finish_reasons` | Array containing the agent stop reason |
167167

168+
## Outcome span attributes
169+
170+
Outcome evaluation data is emitted in OpenTelemetry spans after safe outputs are checked against repository state. These attributes are the span-level view of the outcomes model described in [Outcomes](/gh-aw/reference/outcomes/).
171+
172+
Workflow-level outcome rollups appear on outcome summary or job conclusion spans. The table below is a high-level, non-exhaustive subset.
173+
174+
| Attribute | Description |
175+
| --- | --- |
176+
| `gh-aw.outcome.total` | Total evaluated outcomes for the run |
177+
| `gh-aw.outcome.accepted` | Number of accepted outcomes |
178+
| `gh-aw.outcome.rejected` | Number of rejected outcomes |
179+
| `gh-aw.outcome.pending` | Number of pending outcomes |
180+
| `gh-aw.outcome.ignored` | Number of ignored outcomes |
181+
| `gh-aw.outcome.acceptance_rate` | Accepted fraction for the evaluated outcomes |
182+
| `gh-aw.outcome.waste_rate` | Rejected fraction for the evaluated outcomes |
183+
184+
Per-item outcome spans can also carry more detailed fields such as object type, URL, comments, review activity, and zero-touch acceptance.
185+
168186
## Trace files and artifacts
169187

170188
When observability is enabled, trace data is also mirrored

‎docs/src/content/docs/reference/outcomes.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
---
2+
title: Outcomes
3+
description: Reference for outcome concepts in GitHub Agentic Workflows, including accepted outcomes, outcome states, and how outcome data relates to cost and observability.
4+
sidebar:
5+
order: 297
6+
---
7+
8+
Outcomes describe what happened after a [safe output](/gh-aw/reference/safe-outputs/) landed in a repository. Safe outputs record what a workflow did. Outcomes record the repository state that can be observed afterward.
9+
10+
For example, a pull request can be merged or closed, an issue can remain relevant or be dismissed, and a comment can lead to follow-up activity or be ignored. Outcome data is based on repository state, not on the workflow's self-assessment.
11+
12+
This page defines the common outcome states, summarizes what `accepted` means across safe output types, and lists the telemetry and cost rollups built from that data.
13+
14+
## Outcome Efficiency
15+
16+
Token and cost data are necessary, but they are not enough. A workflow can become cheaper because it became more efficient, or because it simply did less useful work. Outcomes make that difference visible by relating effective tokens to accepted results.
17+
18+
Outcome efficiency is measured as effective tokens divided by accepted outcomes. Lower is better: a lower value means the workflow spent less effective AI work per accepted result.
19+
20+
## Outcome States
21+
22+
To support that measurement, every evaluated output is classified into an outcome state. These states provide the base vocabulary for the rest of the page.
23+
24+
| Outcome | Meaning |
25+
| --- | --- |
26+
| `accepted` | The result was kept, merged, completed, or otherwise accepted by the repository state. |
27+
| `rejected` | The result was explicitly undone, closed, removed, or not accepted. |
28+
| `pending` | The result exists, but has not reached a terminal state yet. |
29+
| `ignored` | The result received no meaningful follow-up within the evaluation window. |
30+
| `noop` | The output type is intentionally non-actionable for outcome measurement. |
31+
32+
Some evaluation systems also distinguish lifecycle-oriented states such as bot-driven cleanup or closure. This page keeps the top-level reference to the common states; see the safe-output outcome specification for the extended lifecycle details.
33+
34+
## Accepted Outcomes
35+
36+
An accepted outcome is the simplest useful unit for measuring workflow effectiveness. Typical examples include merged pull requests, issues that remained relevant and were completed, and labels or comments that stuck and were acted on.
37+
38+
Accepted outcomes are intentionally simpler than a full value model. They do not try to rank one accepted result as inherently more important than another.
39+
40+
> [!NOTE]
41+
> Different output types can have different practical importance. The outcomes model keeps the base measurement simple first. If needed, compare workflows within the same output class before introducing more complex weighting.
42+
43+
The table below is the quick lookup for what `accepted` currently means for each safe output type and whether that meaning comes from a dedicated rule, a fallback rule, a limited check, or no implemented rule yet.
44+
45+
Rows marked `fallback rule` use a generic existence check, not a type-specific rule. For exact rules, edge cases, and conformance details, see [Safe Output Outcome Evaluation Specification](/gh-aw/specs/safe-output-outcome-evaluation/).
46+
47+
Outcome evaluation is based on visible repository state and visible actor identity. A non-bot actor may still be AI-assisted; the lookup reflects what the system can observe, not hidden authoring provenance.
48+
49+
| Safe output type | `accepted` at a glance | Current rule source |
50+
| --- | --- | --- |
51+
| `create_pull_request` | merged | dedicated rule |
52+
| `create_issue` | completed/closed | dedicated rule |
53+
| `add_comment` | reacted to or replied to | dedicated rule |
54+
| `add_labels` | label retention | limited check |
55+
| `add_reviewer` | review target exists | fallback rule |
56+
| `update_issue` | issue still exists | fallback rule |
57+
| `update_pull_request` | PR still exists | fallback rule |
58+
| `close_issue` | still closed | dedicated rule |
59+
| `close_pull_request` | still closed | dedicated rule |
60+
| `close_discussion` | none yet | no implemented rule yet |
61+
| `create_discussion` | none yet | no implemented rule yet |
62+
| `update_discussion` | discussion target exists | fallback rule |
63+
| `create_pull_request_review_comment` | none yet | no implemented rule yet |
64+
| `submit_pull_request_review` | PR still exists | fallback rule |
65+
| `reply_to_pull_request_review_comment` | review target exists | fallback rule |
66+
| `resolve_pull_request_review_thread` | none yet | no implemented rule yet |
67+
| `push_to_pull_request_branch` | merged | dedicated rule |
68+
| `mark_pull_request_as_ready_for_review` | reviewed | dedicated rule |
69+
| `assign_to_agent` | merged or completed | dedicated rule |
70+
| `dispatch_workflow` | dispatch target exists | fallback rule |
71+
| `autofix_code_scanning_alert` | alert target exists | fallback rule |
72+
| `create_code_scanning_alert` | alert target exists | fallback rule |
73+
| `link_sub_issue` | sub-issue link target exists | fallback rule |
74+
| `hide_comment` | none yet | no implemented rule yet |
75+
| `assign_milestone` | milestone still set | dedicated rule |
76+
| `update_project` | project target exists | fallback rule |
77+
| `update_release` | release target exists | fallback rule |
78+
| `noop` | skipped | skipped |
79+
| `missing_tool` | skipped | skipped |
80+
81+
## Telemetry
82+
83+
Outcome data is derived from safe outputs and later checked against repository state. The system records the safe output produced by the workflow, looks up the affected repository object later, and classifies the observed state into an outcome.
84+
85+
This makes outcome evaluation external and observable. The workflow does not decide whether it succeeded; the repository state does.
86+
87+
Outcome information appears in OpenTelemetry spans and related artifacts. Workflow-level rollups such as accepted counts and acceptance rate are emitted on outcome summary or conclusion spans, and per-item spans can carry more detailed fields such as object type, URL, comments, review activity, and zero-touch acceptance.
88+
89+
For the span-level attribute inventory, see [OpenTelemetry](/gh-aw/reference/open-telemetry/).
90+
91+
## Cost and Rollups
92+
93+
Outcomes are most useful when read together with cost data. At the workflow level, the basic questions are how many effective tokens a workflow spent, how many accepted outcomes it produced, and how many effective tokens each accepted outcome cost.
94+
95+
The basic dashboard for outcomes is therefore intentionally small: total effective tokens, total accepted outcomes, effective tokens per accepted outcome, a trend over time, and a workflow ranking by effective tokens per accepted outcome.
96+
97+
For simple workflows, a single run is usually the right unit for outcome measurement.
98+
99+
For orchestrated workflows, multiple runs can belong to one logical execution. In that case, the more meaningful unit is the episode. Outcome and cost totals can be rolled up from runs into episodes using simple sums, and then from episodes into workflow totals and repository totals.
100+
101+
The outcomes model is deliberately narrow. It does not try to estimate the full business value of a workflow, replace human judgment for nuanced quality questions, combine deterministic compute cost and inference cost into one synthetic score, or solve overlap and duplicate-work analysis in the first version.
102+
103+
Those questions may matter later, but they are separate from the base outcomes model described here.
104+
105+
## Related Documentation
106+
107+
- [Cost Management](/gh-aw/reference/cost-management/) explains how workflow cost is measured and reduced.
108+
- [OpenTelemetry](/gh-aw/reference/open-telemetry/) describes the span attributes and artifacts that carry workflow telemetry.
109+
- [Safe Outputs](/gh-aw/reference/safe-outputs/) explains how workflows produce constrained actions.
110+
- [Safe Output Outcome Evaluation Specification](/gh-aw/specs/safe-output-outcome-evaluation/) defines the detailed evaluation logic for each safe output type.

0 commit comments

Comments
 (0)