ci: budget issue-triage prompt under the GitHub Models token cap (#1514)

[SealKan]

Fixes #1514.

Problem

build_prompt assembles up to 6 BM25 candidates (3500 chars each) + the ~92-tool list + a 16000-char body + static framing. Per the issue, the non-body portion alone can reach or exceed the GitHub Models 8000-token input cap, so Evaluate gets an HTTP 413 on essentially every issue. 413 was unhandled (only 429 and >= 500 were), so it broke straight to setFailed, leaving the issue with a triage-failed label. The gpt-4o fallback shares the same cap, so it would 413 too.

Fix

• build_prompt — proactive budgeting (primary fix). After assembly, enforce a conservative token budget (chars/4 estimate, target 7000 to leave margin under the 8000 cap for the 800-token response). Trim in priority order so the actual issue text survives longest: drop BM25 duplicate candidates first, then the tool list, then truncate the body down to a 2000-char floor. The assembled token estimate is logged.
• Evaluate — 413 backstop. Catch 413 specifically and retry once with a truncated prompt instead of breaking to setFailed. With the budgeting above this should not fire; it guards against estimate drift on token-dense bodies (logs/JSON).

Testing

This runs in CI against GitHub Models with no checkout, so the assembly can't import a shared module and can't be exercised in a normal unit test. I added scripts/verify_triage_prompt_budget.mjs, which mirrors the trim algorithm and asserts worst-case inputs (16000-char body + full tool list + 6 max BM25 candidates) stay under budget, the body is never trimmed below its floor, duplicates are dropped first, and small inputs are left untouched. Runs clean with node scripts/verify_triage_prompt_budget.mjs.

As you noted, the end-to-end path can only really be confirmed by merging and opening a fresh test issue. The harness keeps the budgeting logic itself checkable; happy to wire it into CI (or refactor the assembly into a checked-out, importable module) as a follow-up if you'd prefer that over the mirrored copy.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses issue #1514 by implementing a robust token-budgeting strategy for the issue-triage workflow. By proactively trimming prompt components—starting with duplicate candidates and tool lists before truncating the issue body—the system avoids exceeding the GitHub Models input token limit. Additionally, it introduces a fallback mechanism to handle 413 errors gracefully, ensuring that the triage process remains resilient even when encountering unexpectedly large inputs.

Highlights

• Proactive Prompt Budgeting: Implemented a token-budgeting mechanism in the build_prompt process to ensure the assembled prompt stays under the GitHub Models 8000-token input cap, prioritizing the retention of issue body text.
• 413 Error Handling: Added a backstop in the Evaluate function to catch HTTP 413 errors and retry with a truncated prompt, preventing triage failures due to token-dense inputs.
• Verification Script: Introduced a new Node.js script, scripts/verify_triage_prompt_budget.mjs, to validate the trimming algorithm against worst-case input scenarios.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/issue-triage.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new Node.js script, scripts/verify_triage_prompt_budget.mjs, designed to verify that the issue-triage prompt-budgeting algorithm keeps the assembled prompt under the input token limit. The review feedback identifies a potential TypeError in the fitToBudget function if parameters like issueBody, toolSection, or duplicateSection are omitted or passed as null/undefined, and suggests defaulting them to empty strings to ensure robustness.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[SealKan]

Ready for review. All checks green; the one Gemini nit (null-safe section args in the verify harness) is fixed in d005f84 and the thread resolved.

Quick summary of choices:

• Budgeting lives in build_prompt so the prompt is capped before the model call; the 413 branch in Evaluate is a backstop for estimate drift, not the primary fix.
• Trim order is BM25 candidates -> tool list -> body (to a 2000-char floor) so the actual issue text is the last thing sacrificed.
• TOKEN_BUDGET is 7000 (chars/4 estimate) to leave headroom under the 8000 cap plus the 800-token response.

The end-to-end path still can't run in CI without a checkout in this workflow, so the harness mirrors the trim logic rather than importing it. If you'd rather have it truly shared, I can add a checkout + move the assembly into an importable module as a follow-up - your call.

[Patch76]

This fixes the right root cause for #1514 — the fixed overhead (framing + BM25 candidates + tool list) was already at the cap before the body, and trimming in priority order (duplicates → tool list → body-to-floor) keeps the actual issue text longest. Input handling stays on the process.env path and the 413 retry is a sensible net.

Two optional, non-blocking notes:

The body-floor branch doesn't re-assert the budget. After issueBody is clamped to BODY_FLOOR, the prompt isn't re-checked against TOKEN_BUDGET. Today that can't breach the cap — the fixed framing is a small fraction of the budget, so framing + floor is far under 8000 — so it only matters if the static framing later grows several-fold, where the prompt would ship over-cap and lean entirely on the 413 retry (a mid-content substring, not a clean trim). A one-line core.warning when the post-floor estimate is still over budget would make the 413 path's "this should not fire" comment self-enforcing.

The harness's floor assertion doesn't exercise the floor. In verify_triage_prompt_budget.mjs, check #2 runs with a small staticText, so the body lands at the overflow-derived length (~21000 chars), not the 2000-char floor — >= BODY_FLOOR passes without driving the body to the Math.max clamp. A case with a large staticText would actually test that path. Since the harness isn't run in CI, the "keep the two in sync" note is also manual-only.

Neither blocks — the change solves #1514 as-is. Both are optional hardening on the triage-process workflow, so @kingpanther13, your call whether either is worth folding in here or leaving for a follow-up.

[kingpanther13]

Thanks for this — the diagnosis is spot-on and the emphasis is right: proactive budgeting as the primary fix with the 413 catch as a backstop, rather than relying on retry alone. The const→let + assemble() closure refactor is correct, the change is security-clean (no new triggers, permissions, or secrets; issue content stays prompt data), and shipping a verification harness for a checkout-less CI step is genuinely appreciated. I ran node scripts/verify_triage_prompt_budget.mjs and it passes as described.

Two things I'd like to see addressed before merge. Both are small and both directly strengthen the guarantee this PR is making.

1. The budgeter never trims changelog, so the "under the cap" guarantee has a gap.

The trim block drops duplicateSection → toolSection → truncates issueBody to a floor, but changelog (the "Recent Release Context" section, issue-triage.yml:635) is left untouched. changelog is a variable input — up to ~20 unreleased commit subjects plus 3x head -50 of CHANGELOG.md (issue-triage.yml:519-532) — which can plausibly reach ~15-21k+ chars for verbose semantic-release entries. Once it does, the assembled prompt stays over the 7000-token target even after every other section is trimmed to the floor, and falls through to the 413 path: the exact failure this PR fixes, just at a higher input threshold.

The harness doesn't catch this because its worst case uses a fixed x(7000) stand-in for static framing and doesn't model changelog at all, so a green run doesn't actually exercise the real worst case.

Suggestion: add changelog as a fourth, lower-priority trim target (truncate to a floor, after duplicates/tools and before or alongside the body), and add a changelogText term to the harness so the worst-case assertion reflects the real untrimmable load.

2. The verification harness isn't wired into CI.

scripts/verify_triage_prompt_budget.mjs is referenced nowhere except the comment in the workflow, so nothing runs it on PRs and nothing enforces that the two copies stay in sync (TOKEN_BUDGET, BODY_FLOOR, drop order). Since it's a deliberate copy of logic that can't be imported (checkout-less workflow), that drift is the precise risk it exists to guard against. The repo already runs Node test harnesses in CI (pr.yml installs node; tests/js/harness.mjs; subprocess wrappers under tests/src/unit/), so this fits an existing pattern. A one-line node scripts/verify_triage_prompt_budget.mjs step (or a small pytest subprocess wrapper) would close it. Optional extra guard: have the harness read the constants and drop-order out of the workflow file and assert they match.

Non-blocking suggestions:

• After all trims, if the prompt is still over budget (the body floor clamped it), only the routine info line is logged. A core.warning naming the 413 backstop would make that path visible instead of silent (same root cause as #1).
• Two harness assertions are currently vacuous: "body never trimmed below floor" passes without the floor ever binding (the huge-body case lands at ~21k chars, far above the 2k floor), and "duplicates dropped first" never forces the tool drop, so it doesn't actually prove ordering. Adding a case that clamps the body to exactly BODY_FLOOR, and one that forces both drops while asserting the body survives intact, would make them test what they claim. The "worst case" case also resolves at ~5750 tokens with the body untouched, so the label is a little optimistic.
• prompt.substring(0, 24000) in the 413 retry is a bare constant decoupled from the cap — a one-line comment (~6000 tokens at chars/4, under the 8000 cap) would document the relationship.

None of the 413 backstop concerns are silent failures — a double-413 or a truncated-prompt garbage response degrades to the existing triage-failed path via continue-on-error and the guarded JSON.parse, which is correct behavior.

[SealKan]

Thanks @Patch76, folded both in (d005f84 was the Gemini nit; this is the new commit):

1. Body-floor re-check. Added a core.warning after the floor clamp when the estimate is still over budget. As you said, today it can't fire — framing is a small fraction of the budget — but it makes the 413 retry's "this should not fire" comment self-enforcing if the static framing ever grows several-fold, instead of silently shipping over-cap.

2. Harness floor coverage. You're right that check fix: documentation formatting and accuracy improvements #2's small staticText let the body land at the overflow-derived ~21000 chars, so >= BODY_FLOOR passed without hitting the clamp. Added a case with a large staticText (framing alone over budget) that drives the body to exactly BODY_FLOOR, asserting === BODY_FLOOR so the Math.max path is actually exercised.

Both are workflow-only hardening, no behavior change for normal-sized issues. Left the shared-module refactor (checkout + import) as a possible follow-up rather than folding it in here.

[SealKan]

Thanks for the thorough review @kingpanther13 — both blockers addressed in 468b85c.

1. changelog is now a trim target. You're right that it was the hole in the guarantee — changelog is variable input and could keep the prompt over budget after everything else was floored, just re-triggering the 413 at a higher threshold. Added it as a third tier between the tool list and the body: drop duplicates → drop tools → truncate release context to a 1500-char floor → truncate body to its floor. The harness now models a changelog term, and its worst case includes a 21000-char release block so a green run exercises the real untrimmable load (previously the x(7000) static stand-in didn't model it at all).

2. Harness wired into CI. Added tests/src/unit/test_triage_prompt_budget.py — it runs node scripts/verify_triage_prompt_budget.mjs (matching the existing node-in-unit-tests pattern; pr.yml already installs node) and, per your optional guard, parses TOKEN_BUDGET/BODY_FLOOR/CHANGELOG_FLOOR out of both the workflow and the harness and asserts they match — so the two copies can't silently drift.

On the non-blocking notes: the body-floor core.warning and the exact-BODY_FLOOR clamp test you mentioned actually went in just before your review (3d938f4) — your review looks like it was against d005f84. This commit also fixes the vacuous assertions you flagged: the worst case is now genuinely over budget so trimming runs, there's a case that forces both the duplicate and tool drops while asserting the body survives intact, and the 24000-char 413 constant now has a comment noting it's ~6000 tokens at chars/4, under the cap.

Local: harness 8/8, test_triage_prompt_budget.py 2/2, ruff clean, workflow YAML + build_prompt JS parse-check clean.

[kingpanther13]

Re-reviewed at 468b85c — both blockers are resolved, and I verified them rather than eyeballing:

• changelog trim: confirmed the new tier (duplicates → tools → release context @1500 floor → body @2000 floor), changelog correctly made mutable, header preserved. Harness 8/8 and the worst case now genuinely runs the full chain.
• CI wiring: ran test_triage_prompt_budget.py (2/2), and confirmed the drift guard actually fails on drift (mutated CHANGELOG_FLOOR → test failed as expected). Verified pr.yml's Unit Tests job installs node and runs tests/src/unit/, so it executes in CI. The non-blocking notes (body-floor warning, the real === BODY_FLOOR clamp test, the both-drops ordering test, the 24000 comment) are all in too — thanks.

One thing left before this can go green: Ruff Lint is failing. ruff format --check wants tests/src/unit/test_triage_prompt_budget.py reformatted — the @pytest.mark.skipif(...) decorator should collapse to a single line. uv run ruff format tests/src/unit/test_triage_prompt_budget.py fixes it. Once CI is green I'm happy to approve.

Optional, your call: the drift guard checks the constants but not the trim drop-order, so a reorder in one file but not the other would slip through. Not blocking.

[Patch76]

I finally got around to overhauling my PR-review setup, and going back over this PR with it turned up two more gaps worth flagging before the budgeting fix lands — both about the prompt's full input surface and fallback, not just the three sections the trim loop touches.

authorSection is unbounded and untrimmed — same class as the changelog point above. The trim block only bounds duplicateSection → toolSection → issueBody, but assemble() also splices in authorSection (:631), built from author_comments, which is concatenated from all of the author's follow-up comments at issue-triage.yml:243-252 with no length bound. So even after changelog is trimmed as suggested, an issue with long or numerous author comments can push the assembled prompt back over the 7000-token budget the trim logic is meant to enforce. Rather than add inputs to the trim chain one at a time, it's probably cleaner to bound every variable assemble() input; keywordSummary is the one that's safe to leave alone (already capped at the top-5 keywords).

The 413 backstop can trade a 413 for a malformed-response failure. On a 413 the retry sends prompt.substring(0, 24000) (:775) — a head truncation. A prompt that actually 413s is well past the cap (>32k chars), and jsonSchema is the last element of assemble() (:690), so cutting at 24000 chars drops the rubric tail and the JSON-schema directive entirely. The retry then asks the model to triage with no output contract, and the downstream JSON.parse of the response has nothing valid to parse. Truncating the issue-body section specifically (keeping framing + rubric + schema) would keep the backstop from breaking the response format.

[SealKan]

Fixed in 6adfc7c — the ruff blocker plus both gaps Patch76 flagged.

@kingpanther13 — Ruff Lint: ran ruff format, the @pytest.mark.skipif decorator is now one line; ruff format --check passes locally. Also folded in your optional note: the drift guard now asserts the section trim order, not just the constants (test_trim_order_in_sync parses the drop sequence from both files against a canonical [duplicate, tool, changelog, author, body]), so a reorder in one file but not the other fails CI.

@Patch76 — both gaps:

1. authorSection unbounded. You're right, it was the same class as changelog. Rather than keep adding sections one at a time, I bounded every variable assemble() input: authorSection is now mutable and joins the trim chain (duplicates → tools → changelog → author comments → body, each secondary section to a floor; keywordSummary stays as-is since it's already capped at top-5). The harness models it now and the worst case includes a 30k-char author block, so a green run exercises it.

2. 413 head-truncation dropped the schema. Fixed properly: assemble() now takes section overrides, build_prompt emits a minimal_prompt (framing + rubric + JSON schema + a BODY_FLOOR-clamped body, secondary sections dropped), and the 413 retry sends that instead of prompt.substring(0, 24000). Since the schema is the last element of assemble(), the old head cut killed the output contract; the minimal prompt keeps it intact so downstream JSON.parse still has something valid.

Local: harness 10/10, test_triage_prompt_budget.py 3/3, ruff check + format clean, workflow YAML + all 8 inline scripts parse-check clean.

[kingpanther13]

Re-reviewed at 6adfc7c. The code itself is in good shape and I verified it rather than eyeballing: harness 10/10, test_triage_prompt_budget.py 3/3, ruff format --check + ruff check clean. I also confirmed the new order guard actually works — swapping the author/body trim lines in the harness makes test_trim_order_in_sync fail as intended. The authorSection bounding, the assemble() override + minimalPrompt 413 backstop (schema preserved, so downstream JSON.parse survives), and the trim-order drift guard all look correct.

The blocker now is outside the PR's own logic: master picked up 6f47c925 ("drop tool list from evaluate prompt + tighten caps to fit 8K token limit") while this was iterating, and it overlaps heavily with this PR:

• It conflicts — both rewrite the build_prompt step, so the PR is currently CONFLICTING/DIRTY.
• That's also why there are no CI checks on 6adfc7c5: pull_request checks run against the merge ref, which GitHub can't build while the merge is dirty. A rebase will bring CI back.
• 6f47c925 removes the tool list entirely (TOOL_LIST/toolList/toolSection), but this PR still carries tool-list trim logic (the toolSection drop tier, the harness tool modeling, and tool in the canonical trim order + markers + worst case). Those need dropping on rebase.

Since 6f47c925 is a competing fix for the same #1514 (static tighter caps vs. this PR's dynamic budgeting + 413 backstop), it'd be worth checking with @julienld (as author of 6f47c925) on whether the dynamic-budgeting approach is still wanted on top of it before you invest in the rebase. If it is, the body cap that 6f47c925 lowered to 5000 chars could likely go back up, since the budgeter now trims dynamically when over budget.

Happy to re-review once it's rebased and CI is green.

(One tiny nit for whenever you next touch it: the author-trim core.info logs only the original length, while the changelog/body logs show X -> Y chars.)

[SealKan]

Thanks for digging into the conflict @kingpanther13 — you're right that 6f47c92 landed the same target while this was iterating, and I'd rather not rebase into a redundant PR.

@julienld — your fix already gets the immediate 413 by tightening the static caps (tool list out, body 16000→5000, candidates 3×600). Before I rebase this on top, worth deciding whether the dynamic approach here still earns its place. What it adds beyond the static caps:

1. Closes two 413 holes your caps don't cover. changelog (the Recent Release Context — up to ~20 unreleased subjects + 3× head -50 of CHANGELOG.md) and authorSection (all author follow-up comments, concatenated unbounded) are still untrimmed after 6f47c92, so a verbose-changelog or long-comment-thread issue can still push the prompt over 8K. This PR bounds every variable section, not just body + candidates.
2. A 413 backstop — if the estimate is ever wrong on token-dense content, it retries once with a minimal prompt (framing + rubric + schema + floored body) instead of failing to triage-failed.
3. Dynamic trimming keeps more context when there's room — body only gets cut when actually over budget, so the common small-issue case keeps full text rather than a hard 5000-char cap.

If you'd like the dynamic budgeting on top, I'll rebase: drop all the tool-list logic (gone in your commit), and — since the budgeter trims dynamically — the 5000-char body cap can likely go back up. If you'd rather keep it simple and static, I'm happy to close this as superseded; no attachment. Your call as author of the competing fix.

(And thanks for the nit re: the author-trim log only showing the original length — I'll fix it to X -> Y chars on rebase if this moves forward.)

[julienld]

@SealKan / @kingpanther13 : Sorry, I saw that the triage bot was still offline so I pushed some quick fixes. I will take a look this weekend. Thanks.

[julienld]

@SealKan — please go ahead and rebase, we want the dynamic approach.

A couple of context updates since your last message:

1. The immediate 413s are unblocked — I pushed two hotfixes directly to master that cleared the queue of triage-failed issues: removed the maintainer check, dropped the tool list from the evaluate prompt, and tightened the static caps. That bought time, but your dynamic approach is the right long-term fix — changelog and authorSection are still untrimmed in the static fix, and the minimalPrompt backstop is cleaner than our current head-truncation.

2. We've also activated a $20/month GitHub Models budget, which should raise the per-request cap above 8K. We don't have the exact new limit yet — worth testing after merge. Either way, keeping the dynamic trimmer is still the right call for correctness.

3. I've given you write access to https://github.com/homeassistant-ai/ha-mcp-workflows-dev — it's the workflow testing environment where we run end-to-end triage simulations by opening issues and triggering workflow_dispatch. Feel free to push your rebased workflow there and test against it before the final PR push.

For the rebase:

• Drop the tool-list trim tier (toolSection drop, harness tool modeling, tool in the canonical order) — it's already gone from master
• The 5000-char body cap and 3×600 candidates can go back up — something like 10K body and 4×1200 candidates would be more generous for the common case since the dynamic trimmer handles it
• Fix the author-trim log to show X → Y chars (you already flagged this)

Thanks for the thorough iteration on this — the assemble() override + minimalPrompt backstop and the trim-order drift guard in CI are both solid additions.

[SealKan]

Rebased onto master (past 6f47c92) and force-pushed — the PR is now a single focused commit layering dynamic budgeting on top of your static caps.

What changed per your guidance:

• Dropped the tool-list trim tier entirely — toolSection, the harness tool modeling, and tool in the canonical trim order are all gone (it's already out of build_prompt on master).
• Raised the caps back up — body 5000 -> 10000, BM25 candidates 3x600 -> 4x1200. The dynamic trimmer drops/floors these only when a prompt actually exceeds budget, so the common case keeps the more generous context.
• Bounded the gaps your static fix leaves open — changelog and authorSection are now trimmed to floors (1500 chars each) in priority order: duplicates dropped -> changelog -> author -> body. A token-dense issue with a verbose release context or long follow-up thread can no longer push past the cap.
• minimalPrompt backstop wired into the new 413 branch in Evaluate — if the estimate ever drifts and the model still 413s, it retries once with framing + rubric + schema + a floored body instead of setFailed. Cleaner than head-truncation, which would cut the trailing JSON schema off and break the output contract.
• Fixed the author-trim log to show X -> Y chars like the changelog/body logs.

Verification (local):

• node scripts/verify_triage_prompt_budget.mjs — 10/10 (worst case = full framing + 4x1200 candidates + 16K body + 21K changelog + 30K author all fit)
• pytest src/unit/test_triage_prompt_budget.py — 3/3 (harness passes, budget constants in sync, trim order in sync)
• ruff format --check + ruff check clean

The drift guard now asserts the canonical order duplicate -> changelog -> author -> body across both files, so the workflow and the mirrored harness can't silently diverge.

Thanks for the write access to ha-mcp-workflows-dev — happy to run an end-to-end workflow_dispatch simulation there before final merge if you'd like a live check. And good to know about the $20/month Models budget; the 7000-token target leaves headroom regardless, but worth re-measuring the real cap once it's live.

[kingpanther13]

Approving. The rebase is clean (tool-list tier dropped, caps raised to body 10k / candidates 4×1200, changelog + authorSection bounded, minimalPrompt 413 backstop), CI is green, and I validated it end-to-end in the workflows-dev environment via workflow_dispatch:

• Normal and large (10k-char body) issues assemble to ~3.1k / ~5.5k tokens — no trimming, no 413, valid triage.
• A token-dense case (large author comment) drove the prompt over budget and the trim chain fired in order: BM25 candidates dropped → release context 4400→1500 → author comments 63841→11629 → landed at exactly ~7000 tokens, under the 8000 cap, no 413, valid structured result. The issue body was never touched, as intended.

One trivial leftover I'll clean up in a fast follow-up (not worth gating this on): the // gpt-4o-mini: 16K input token free-tier limit ... comment in the Evaluate step is the stale assumption #1514 names as root cause, and it now contradicts this PR's own 8000-cap budgeting. I'll also fold in two small test-hardening assertions (pin the harness's static-framing assumption; an author-floor boundary test). All non-blocking.

Thanks for the thorough iteration — the assemble() override + minimalPrompt backstop and the trim-order drift guard are great additions.