Skip to content

Commit 350dfb3

Browse files
CopilotCopilot
authored
feat: add "Optimize token consumption" section to agent failure issues when a guardrail triggers (#39069)
1 parent 61344a1 commit 350dfb3

9 files changed

Lines changed: 513 additions & 0 deletions

File tree

‎.github/aw/optimize-agentic-workflow.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
---
2+
description: Analyze and reduce token consumption in agentic workflows — audit-based measurement, DataOps, gh-proxy, sub-agents, and prompt optimization.
3+
disable-model-invocation: true
4+
---
5+
6+
# Agentic Workflow Token Optimizer
7+
8+
Help users reduce the AI token usage and cost of GitHub Agentic Workflows in this repository.
9+
10+
## Load These References First
11+
12+
- [github-agentic-workflows.md](github-agentic-workflows.md)
13+
- [token-optimization.md](token-optimization.md)
14+
- [workflow-editing.md](workflow-editing.md)
15+
- [syntax.md](syntax.md)
16+
17+
Load these only when relevant:
18+
19+
- [experiments.md](experiments.md)
20+
- [safe-outputs.md](safe-outputs.md)
21+
22+
## Available Commands
23+
24+
```bash
25+
gh aw audit <run-id> --json
26+
gh aw audit <base-run-id> <optimized-run-id>
27+
gh aw logs <workflow-name> --json
28+
gh aw compile <workflow-name>
29+
gh aw status
30+
```
31+
32+
## Start the Conversation
33+
34+
Ask for one of these inputs:
35+
36+
- a workflow run URL (or run ID) to analyze
37+
- a workflow name to review the source
38+
- the guardrail that was exceeded (max-ai-credits, max-daily-ai-credits, max-tool-denials, max-turns / timeout)
39+
40+
## Fast Path: Run URL Provided
41+
42+
If the user gives a GitHub Actions run URL:
43+
44+
1. Extract the run ID
45+
2. Run `gh aw audit <run-id> --json`
46+
3. Inspect `agent_usage.aic`, `agent_usage.input_tokens`, `agent_usage.output_tokens`, `agent_usage.cache_read_tokens`
47+
4. Identify the most expensive phases before asking additional questions
48+
49+
## Guardrail-Specific Entry Points
50+
51+
### `max-ai-credits` exceeded
52+
53+
The workflow was stopped because it consumed more AI Credits than the configured per-run budget.
54+
55+
Priority checks:
56+
1. Which tool calls dominated token usage? (`token-usage.jsonl`)
57+
2. Is the prompt front-loading large payloads that could be fetched on demand?
58+
3. Are there repetitive extraction steps that sub-agents could handle cheaply?
59+
4. Does the frontier model handle tasks that a small model could do?
60+
61+
### `max-daily-ai-credits` exceeded
62+
63+
The workflow is being blocked because its 24-hour AI Credits budget is exhausted.
64+
65+
Priority checks:
66+
1. What is the run cadence? (scheduled too frequently?)
67+
2. Does the workflow use cheap triage before escalating to the frontier model?
68+
3. Is batching or caching applicable to reduce run frequency?
69+
4. Are there noop early-exits for events that do not require agent action?
70+
71+
### `max-tool-denials` exceeded
72+
73+
The Copilot SDK hit the tool-denial threshold, indicating the prompt attempted actions outside the allowed tool policy.
74+
75+
Priority checks:
76+
1. What tool was repeatedly denied? (last denied reason in the failure issue)
77+
2. Is the tool missing from the workflow's permissions/firewall config?
78+
3. Can the prompt be revised to avoid the denied operation entirely?
79+
4. Would a DataOps pre-step satisfy the data need without a tool call?
80+
81+
### Timeout / `max-turns` exceeded
82+
83+
The agent ran out of time or turns before completing the task.
84+
85+
Priority checks:
86+
1. Is the task decomposable into smaller, faster sub-tasks?
87+
2. Are there long-running tool calls that could be replaced with DataOps pre-steps?
88+
3. Is the prompt asking the agent to do too much in one run?
89+
4. Can `max-turns` or `timeout-minutes` be raised, or should the task be split?
90+
91+
## Optimization Analysis Plan
92+
93+
After measuring token usage, produce a prioritized plan:
94+
95+
1. **Measure** — run `gh aw audit <run-id>` and summarize AI Credits and per-call token breakdown
96+
2. **Identify top cost drivers** — list the three most expensive phases/tool calls
97+
3. **Apply quick wins first** — DataOps pre-steps, `gh-proxy`, `cli-proxy`, prompt trimming
98+
4. **Sub-agent delegation** — identify repetitive per-item loops suitable for small-model workers
99+
5. **Prompt caching** — verify stable context appears before dynamic content
100+
6. **Experiment** — add an `experiments:` entry with `metric: "aic"` to measure the change
101+
7. **Validate quality** — confirm the optimized run produces equivalent safe outputs
102+
103+
Present the plan clearly before making any edits. Confirm with the user before applying changes.
104+
105+
## Editing Workflow
106+
107+
1. Edit `.github/workflows/<workflow-name>.md`
108+
2. Recompile: `gh aw compile <workflow-name>`
109+
3. Commit both the source and the generated `.lock.yml`
110+
4. Report the estimated savings and link to the PR or commit

‎.github/skills/agentic-workflows/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ Load these files from `github/gh-aw` (they are not available locally).
3232
- `.github/aw/memory.md`
3333
- `.github/aw/messages.md`
3434
- `.github/aw/network.md`
35+
- `.github/aw/optimize-agentic-workflow.md`
3536
- `.github/aw/patterns.md`
3637
- `.github/aw/pr-reviewer.md`
3738
- `.github/aw/report.md`

‎.github/skills/optimize-agentic-workflow/SKILL.md‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
---
2+
name: optimize-agentic-workflow
3+
description: Analyze and reduce token consumption in agentic workflows — guardrail-specific entry points, measurement, and optimization techniques.
4+
---
5+
6+
# Agentic Workflow Token Optimizer
7+
8+
Help users reduce the AI token usage and cost of GitHub Agentic Workflows in this repository.
9+
10+
## Load These References First
11+
12+
Load these files from `github/gh-aw` (they are not available locally).
13+
14+
- `.github/aw/github-agentic-workflows.md`
15+
- `.github/aw/token-optimization.md`
16+
- `.github/aw/workflow-editing.md`
17+
- `.github/aw/syntax.md`
18+
19+
Load these only when relevant:
20+
21+
- `.github/aw/experiments.md`
22+
- `.github/aw/safe-outputs.md`
23+
24+
## Available Commands
25+
26+
```bash
27+
gh aw audit <run-id> --json
28+
gh aw audit <base-run-id> <optimized-run-id>
29+
gh aw logs <workflow-name> --json
30+
gh aw compile <workflow-name>
31+
gh aw status
32+
```
33+
34+
## Start the Conversation
35+
36+
Ask for one of these inputs:
37+
38+
- a workflow run URL (or run ID) to analyze
39+
- a workflow name to review the source
40+
- the guardrail that was exceeded (max-ai-credits, max-daily-ai-credits, max-tool-denials, max-turns / timeout)
41+
42+
## Fast Path: Run URL Provided
43+
44+
If the user gives a GitHub Actions run URL:
45+
46+
1. Extract the run ID
47+
2. Run `gh aw audit <run-id> --json`
48+
3. Inspect `agent_usage.aic`, `agent_usage.input_tokens`, `agent_usage.output_tokens`, `agent_usage.cache_read_tokens`
49+
4. Identify the most expensive phases before asking additional questions
50+
51+
## Guardrail-Specific Entry Points
52+
53+
### `max-ai-credits` exceeded
54+
55+
The workflow was stopped because it consumed more AI Credits than the configured per-run budget.
56+
57+
Priority checks:
58+
1. Which tool calls dominated token usage? (`token-usage.jsonl`)
59+
2. Is the prompt front-loading large payloads that could be fetched on demand?
60+
3. Are there repetitive extraction steps that sub-agents could handle cheaply?
61+
4. Does the frontier model handle tasks that a small model could do?
62+
63+
### `max-daily-ai-credits` exceeded
64+
65+
The workflow is being blocked because its 24-hour AI Credits budget is exhausted.
66+
67+
Priority checks:
68+
1. What is the run cadence? (scheduled too frequently?)
69+
2. Does the workflow use cheap triage before escalating to the frontier model?
70+
3. Is batching or caching applicable to reduce run frequency?
71+
4. Are there noop early-exits for events that do not require agent action?
72+
73+
### `max-tool-denials` exceeded
74+
75+
The Copilot SDK hit the tool-denial threshold, indicating the prompt attempted actions outside the allowed tool policy.
76+
77+
Priority checks:
78+
1. What tool was repeatedly denied? (last denied reason in the failure issue)
79+
2. Is the tool missing from the workflow's permissions/firewall config?
80+
3. Can the prompt be revised to avoid the denied operation entirely?
81+
4. Would a DataOps pre-step satisfy the data need without a tool call?
82+
83+
### Timeout / `max-turns` exceeded
84+
85+
The agent ran out of time or turns before completing the task.
86+
87+
Priority checks:
88+
1. Is the task decomposable into smaller, faster sub-tasks?
89+
2. Are there long-running tool calls that could be replaced with DataOps pre-steps?
90+
3. Is the prompt asking the agent to do too much in one run?
91+
4. Can `max-turns` or `timeout-minutes` be raised, or should the task be split?
92+
93+
## Optimization Analysis Plan
94+
95+
After measuring token usage, produce a prioritized plan:
96+
97+
1. **Measure** — run `gh aw audit <run-id> --json` and summarize AI Credits and per-call token breakdown
98+
2. **Identify top cost drivers** — list the three most expensive phases/tool calls
99+
3. **Apply quick wins first** — DataOps pre-steps, `gh-proxy`, `cli-proxy`, prompt trimming
100+
4. **Sub-agent delegation** — identify repetitive per-item loops suitable for small-model workers
101+
5. **Prompt caching** — verify stable context appears before dynamic content
102+
6. **Experiment** — add an `experiments:` entry with `metric: "aic"` to measure the change
103+
7. **Validate quality** — confirm the optimized run produces equivalent safe outputs
104+
105+
Present the plan clearly before making any edits. Confirm with the user before applying changes.
106+
107+
## Editing Workflow
108+
109+
1. Edit `.github/workflows/<workflow-name>.md`
110+
2. Recompile: `gh aw compile <workflow-name>`
111+
3. Commit both the source and the generated `.lock.yml`
112+
4. Report the estimated savings and link to the PR or commit

‎actions/setup/js/handle_agent_failure.cjs‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1665,6 +1665,40 @@ function buildDailyAICExceededContext(hasDailyAICExceeded, totalAIC, threshold)
16651665
);
16661666
}
16671667

1668+
/**
1669+
* Build the "Optimize token consumption" details section for the failure issue when a guardrail
1670+
* limit was the root cause of the failure.
1671+
*
1672+
* Guardrails that trigger this section:
1673+
* - max-ai-credits: per-run AI Credits budget exceeded
1674+
* - max-daily-ai-credits: 24-hour per-workflow AI Credits quota exhausted
1675+
* - max-tool-denials: Copilot SDK tool-denial threshold hit
1676+
* - max-turns / timeout: agent ran out of turns or wall-clock time
1677+
*
1678+
* @param {object} options
1679+
* @param {boolean} options.maxAICreditsExceeded - max-ai-credits guardrail triggered
1680+
* @param {boolean} options.hasDailyAICExceeded - max-daily-ai-credits guardrail triggered
1681+
* @param {boolean} options.hasToolDenialsExceeded - max-tool-denials guardrail triggered
1682+
* @param {boolean} options.isTimedOut - timeout / max-turns guardrail triggered
1683+
* @param {string} options.runUrl - URL to the failed workflow run
1684+
* @returns {string} Rendered section or empty string when no guardrail was triggered
1685+
*/
1686+
function buildOptimizeTokenConsumptionContext({ maxAICreditsExceeded, hasDailyAICExceeded, hasToolDenialsExceeded, isTimedOut, runUrl }) {
1687+
const guardrailTriggered = maxAICreditsExceeded || hasDailyAICExceeded || hasToolDenialsExceeded || isTimedOut;
1688+
if (!guardrailTriggered) {
1689+
return "";
1690+
}
1691+
1692+
let guardrailName = "guardrail limit";
1693+
if (maxAICreditsExceeded) guardrailName = "max-ai-credits";
1694+
else if (hasDailyAICExceeded) guardrailName = "max-daily-ai-credits";
1695+
else if (hasToolDenialsExceeded) guardrailName = "max-tool-denials";
1696+
else if (isTimedOut) guardrailName = "max-turns / timeout";
1697+
1698+
const templatePath = getPromptPath("optimize_token_consumption_context.md");
1699+
return renderTemplateFromFile(templatePath, { guardrail_name: guardrailName, run_url: runUrl });
1700+
}
1701+
16681702
// Maps engine ID (GH_AW_ENGINE_ID) to credential name for use with GH_AW_ENGINE_API_HOSTS.
16691703
const ENGINE_ID_TO_CREDENTIAL = /** @type {Record<string, string>} */ {
16701704
copilot: "`COPILOT_GITHUB_TOKEN`",
@@ -3116,6 +3150,9 @@ async function main() {
31163150
// Build credential auth error context (firewall audit.jsonl 401/403 from provider endpoints)
31173151
const credentialAuthErrorContext = buildCredentialAuthErrorContext();
31183152

3153+
// Build optimize token consumption context (shown when a guardrail was the failure root cause)
3154+
const optimizeTokenConsumptionContext = buildOptimizeTokenConsumptionContext({ maxAICreditsExceeded, hasDailyAICExceeded, hasToolDenialsExceeded, isTimedOut, runUrl });
3155+
31193156
// Create template context with sanitized workflow name
31203157
const templateContext = {
31213158
workflow_name: sanitizedWorkflowName,
@@ -3151,6 +3188,7 @@ async function main() {
31513188
lockdown_check_failed_context: lockdownCheckFailedContext,
31523189
stale_lock_file_failed_context: staleLockFileFailedContext,
31533190
daily_ai_credits_exceeded_context: dailyAICExceededContext,
3191+
optimize_token_consumption_context: optimizeTokenConsumptionContext,
31543192
};
31553193

31563194
// Render the issue template
@@ -3234,6 +3272,7 @@ module.exports = {
32343272
buildLockdownCheckFailedContext,
32353273
buildStaleLockFileFailedContext,
32363274
buildDailyAICExceededContext,
3275+
buildOptimizeTokenConsumptionContext,
32373276
buildTimeoutContext,
32383277
shouldBuildEngineFailureContext,
32393278
isIssueWritePermissionError,

0 commit comments

Comments
 (0)