Skip to content

fix(daemon): guard model context before launch - #5815

Open
Siri-Ray wants to merge 22 commits into
mainfrom
agent/model-context-budget
Open

fix(daemon): guard model context before launch#5815
Siri-Ray wants to merge 22 commits into
mainfrom
agent/model-context-budget

Conversation

@Siri-Ray

Copy link
Copy Markdown
Contributor

Why

The 0.15.0 reliability review found agent runs reaching provider context-window limits only after launch, leaving users with a slow and opaque failure. This PR adds a daemon-side preflight so known model limits fail fast and the decision is observable in both PostHog and Langfuse.

It also preserves context/output limits returned by AMR/OpenCode and provider model catalogs instead of discarding that metadata.

What users will see

Oversized prompts for models with a known context limit now fail before the agent CLI starts, with an actionable AGENT_PROMPT_TOO_LARGE error. Models whose limit is not known continue unchanged; the guard is observation-only for them.

Surface area

  • UI — new page / dialog / panel / menu item / setting / empty state in apps/web or apps/desktop (including Electron menu bar)
  • Keyboard shortcut — new or changed
  • CLI / env var — new od subcommand or flag, new tools-dev / tools-pack / tools-pr flag, or new OD_* env var
  • API / contract — new /api/* endpoint, new SSE event, or changed shape in packages/contracts
  • Extension point — new entry under skills/, design-systems/, design-templates/, or craft/, or change to the skills protocol
  • i18n keys — added new translation keys (see TRANSLATIONS.md for the locale workflow)
  • New top-level dependency — adding any new entry to the root package.json (dependencies or devDependencies); workspace-package package.json files are out of scope. Include a paragraph on what we get vs. what bytes we ship (see CONTRIBUTING.md → Code style)
  • Default behavior change — changes what existing users experience without opting in (default model, default setting, file/SQLite schema, auto-network on startup, auto-install)
  • None — internal refactor, docs, tests, or translation update only

Screenshots

Not applicable; there is no new UI surface.

Bug fix verification

  • Test paths: apps/daemon/tests/runtimes/model-context-budget.test.ts, apps/daemon/tests/runtimes/run-failure-telemetry-smoke.test.ts, and apps/daemon/tests/amr-acp-integration.test.ts.
  • The new specs failed before the implementation (missing budget module/catalog fields) and pass on this branch.
  • The smoke test exercises the real daemon run endpoint and verifies the oversized request is blocked before the fake CLI can emit its configured authentication error.

Validation

  • pnpm typecheck
  • pnpm --filter @open-design/daemon exec vitest run tests/runtimes/model-context-budget.test.ts tests/amr-acp-integration.test.ts tests/langfuse-trace.test.ts (114 passed)
  • pnpm --filter @open-design/contracts exec vitest run tests/analytics-run-finished-contract.test.ts (4 passed)
  • pnpm --filter @open-design/daemon exec vitest run tests/runtimes/run-failure-telemetry-smoke.test.ts (4 passed)
  • pnpm guard reaches the cross-app manifest check and then fails on the current base because apps/telemetry-worker/package.json is absent; all preceding and subsequent reported checks pass.

@lefarcen
lefarcen requested a review from nettee July 17, 2026 10:19
@lefarcen lefarcen added size/L PR changes 300-700 lines risk/high High risk: apps/desktop, daemon, auth, migration, workflows, package deps type/bugfix Bug fix labels Jul 17, 2026
@Siri-Ray
Siri-Ray marked this pull request as ready for review July 18, 2026 03:05

@nettee nettee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new preflight budgeting path is close, but one supported Claude-family id shape still bypasses the fallback guard when catalog metadata is missing. See the inline note for the concrete matcher gap and the test case I think should be added.

🔁 Powered by Looper · runner=reviewer · agent=codex · An autonomous AI dev team for your GitHub repos.

Comment thread apps/daemon/src/runtimes/model-context-budget.ts Outdated
@lefarcen lefarcen added the needs-validation Runtime change detected; needs human or /explore agent validation. label Jul 18, 2026
@lefarcen

Copy link
Copy Markdown
Contributor

🧪 Heads-up: this change touches a user-visible run path, so we'll want a manual QA pass before merge. Please hold off merging once the current blocking review is addressed — we'll loop QA in when the PR is otherwise merge-ready.

Generated-By: looper 0.11.0 (runner=fixer, agent=codex)
@Siri-Ray

Copy link
Copy Markdown
Contributor Author

Looper fixer round completecf95984

  • ✅ Review comment on apps/daemon/src/runtimes/model-context-budget.ts (@nettee) — thread
    • Expanded the Claude-family matcher in apps/daemon/src/runtimes/model-context-budget.ts to accept dotted provider prefixes, and added a Bedrock regression in apps/daemon/tests/runtimes/model-context-budget.test.ts.

🔁 Powered by Looper · runner=fixer · agent=codex · An autonomous AI dev team for your GitHub repos.

@nettee nettee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One reachable Claude model path still bypasses the new preflight guard, so I’m keeping this blocked until that gap is closed.

🔁 Powered by Looper · runner=reviewer · agent=codex · An autonomous AI dev team for your GitHub repos.

Comment thread apps/daemon/src/runtimes/model-context-budget.ts
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Visual regression review

Head: 4583df9 · Base: eee0377

29 changed · 20 unchanged · 0 new without baseline · 0 failed

Changed cases

Case Main PR Diff
visual-critical-settings
871 px (0.07%)
main pr diff
visual-home
2,658 px (0.21%)
main pr diff
visual-home-catalog
710 px (0.05%)
main pr diff
visual-home-context-picker
2,313 px (0.18%)
main pr diff
visual-home-context-picker-popover
1,237 px (0.40%)
main pr diff
visual-home-plugin-filter
555 px (0.04%)
main pr diff
visual-home-plugin-use-staged
6,123 px (0.47%)
main pr diff
visual-home-plugin-use-with-query
2,320 px (0.18%)
main pr diff
visual-home-staged-attachment
5,531 px (0.43%)
main pr diff
visual-integrations
521 px (0.04%)
main pr diff
visual-integrations-mcp
521 px (0.04%)
main pr diff
visual-integrations-use-everywhere
871 px (0.07%)
main pr diff
visual-onboarding-cloud
3,566 px (0.28%)
main pr diff
visual-plugin-details
717 px (0.06%)
main pr diff
visual-plugin-share-menu
741 px (0.06%)
main pr diff
visual-plugins
710 px (0.05%)
main pr diff
visual-projects
2,361 px (0.18%)
main pr diff
visual-projects-kanban
2,366 px (0.18%)
main pr diff
visual-settings-byok
1,721 px (0.13%)
main pr diff
visual-settings-byok-model-dropdown
1,721 px (0.13%)
main pr diff

9 additional changed case(s) omitted from this comment.

Unchanged cases
Case Main PR Diff
visual-avatar-local-agent-list
57 px (0.00%)
main pr diff
visual-avatar-local-agent-list-panel
0 px (0.00%)
main pr diff
visual-avatar-menu
167 px (0.01%)
main pr diff
visual-avatar-menu-panel
30 px (0.09%)
main pr diff
visual-avatar-open-design-model-picker
263 px (0.02%)
main pr diff
visual-critical-workspace
374 px (0.03%)
main pr diff
visual-critical-workspace-preview
0 px (0.00%)
main pr diff
visual-design-system-detail
120 px (0.01%)
main pr diff
visual-design-systems
120 px (0.01%)
main pr diff
visual-new-project-modal
0 px (0.00%)
main pr diff
visual-plugin-share-menu-popover
31 px (0.13%)
main pr diff
visual-project-avatar-model-dropdown
57 px (0.00%)
main pr diff
visual-project-avatar-model-dropdown-popover
0 px (0.00%)
main pr diff
visual-project-workspace
495 px (0.04%)
main pr diff
visual-settings-byok-model-dropdown-popover
0 px (0.00%)
main pr diff
visual-settings-local-cli-model-dropdown-popover
0 px (0.00%)
main pr diff
visual-topbar-byok-switcher-popover
0 px (0.00%)
main pr diff
visual-topbar-execution-switcher-popover
0 px (0.00%)
main pr diff
visual-topbar-local-cli-model-list-popover
0 px (0.00%)
main pr diff
visual-workspace-staged-contexts
0 px (0.00%)
main pr diff

Visual diff is advisory only and does not block merging.

Generated-By: looper 0.11.0 (runner=fixer, agent=codex)
@Siri-Ray

Copy link
Copy Markdown
Contributor Author

Looper fixer round complete4c9682f

  • ✅ Review comment on apps/daemon/src/runtimes/model-context-budget.ts:65 (@nettee) — thread
    • Recognized the built-in sonnet, opus, and haiku aliases in model-context-budget.ts and added a modelId: 'sonnet' regression test.

🔁 Powered by Looper · runner=fixer · agent=codex · An autonomous AI dev team for your GitHub repos.

@nettee nettee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One remaining blocker: the AMR OpenCode enrichment path still drops the new context/output metadata for a reachable class of live catalog entries, so the preflight guard can remain unmeasured for models this PR is supposed to fail fast.

🔁 Powered by Looper · runner=reviewer · agent=codex · An autonomous AI dev team for your GitHub repos.

Comment thread apps/daemon/src/runtimes/defs/amr.ts
Generated-By: looper 0.11.0 (runner=fixer, agent=codex)
@Siri-Ray

Copy link
Copy Markdown
Contributor Author

Looper fixer round complete394cbad

  • ✅ Review comment on apps/daemon/src/runtimes/defs/amr.ts:511 (@nettee) — thread
    • Updated enrichVelaModelsFromOpenCodeCatalog() to merge OpenCode metadata for every live Vela model while preserving live pricing. Added a regression covering live deepseek-v4-flash pricing with OpenCode-only context and output limits.

🔁 Powered by Looper · runner=fixer · agent=codex · An autonomous AI dev team for your GitHub repos.

@nettee nettee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two blockers remain in the new context-budget path: the rollover compactor can still hand the launch guard an over-budget prompt, and the resume-side rollover check undercounts turns that resend additional instructions. See the inline notes for the concrete paths.

🔁 Powered by Looper · runner=reviewer · agent=codex · An autonomous AI dev team for your GitHub repos.

Comment thread apps/daemon/src/runtimes/model-context-budget.ts Outdated
Comment thread apps/daemon/src/server.ts Outdated
Budget the exact resumed-turn payload after stable instruction selection, and guarantee transcript tail compaction stays within its token target. Resolve the current main merge conflicts additively across context-budget and retry telemetry.

Generated-By: looper 0.11.0 (runner=fixer, agent=codex)
@Siri-Ray

Copy link
Copy Markdown
Contributor Author

Looper fixer round complete0723666

  • ✅ Review comment on apps/daemon/src/runtimes/model-context-budget.ts (@nettee) — thread
    • Added a UTF-8-safe retained-tail fallback in apps/daemon/src/runtimes/model-context-budget.ts so even one oversized transcript block stays within maxTokens, with a single-block regression test.
  • ✅ Review comment on apps/daemon/src/server.ts (@nettee) — thread
    • Moved resume budgeting in apps/daemon/src/server.ts to the fully composed payload after stable-instruction selection; changed stable hashes now trigger rollover and prompt rebuilding, covered by an OpenCode integration regression.

🔁 Powered by Looper · runner=fixer · agent=codex · An autonomous AI dev team for your GitHub repos.

@nettee nettee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocker remains in the rollover path: the new transcript compaction target can still be larger than the selected model's safe input budget, so some resumable turns still fail with AGENT_PROMPT_TOO_LARGE instead of reseeding cleanly.

🔁 Powered by Looper · runner=reviewer · agent=codex · An autonomous AI dev team for your GitHub repos.

Comment thread apps/daemon/src/server.ts Outdated
@lefarcen

Copy link
Copy Markdown
Contributor

Following @nettee's latest review on the current head, the remaining blocker is now pretty focused: the rollover compaction target in the resume path still needs to be derived from the model's real input budget so low-context models reseed cleanly, along with the regression for that case. Once that lands, this can go back through the normal review/QA path.

💡 To drive this PR to merge hands-free, paste this to your AI coding agent (Claude Code / Codex / opencode / Cursor …):
Take over nexu-io/open-design#5815 until it merges — read https://raw.githubusercontent.com/nexu-io/looper/main/skills/pr-takeover/SKILL.md and follow it.

Account for fresh-session instructions and turn framing before choosing the transcript compaction target, so low-context rollovers can reseed without tripping the launch guard.

Generated-By: looper 0.11.0 (runner=fixer, agent=codex)
@Siri-Ray

Copy link
Copy Markdown
Contributor Author

Looper fixer round completee47bab4

  • ✅ Review comment on apps/daemon/src/server.ts (@nettee) — thread
    • Updated apps/daemon/src/server.ts to subtract fresh-session instructions and turn framing from the input budget before compacting. Added a low-context metadata rollover regression in apps/daemon/tests/opencode-session-resume.test.ts.

🔁 Powered by Looper · runner=fixer · agent=codex · An autonomous AI dev team for your GitHub repos.

@lefarcen

Copy link
Copy Markdown
Contributor

Thanks for the focused follow-up here — the rollover fix and the low-context regression are now on the current head (e47bab4). Re-requesting @nettee on this head so the remaining blocker can get a fresh pass.

@lefarcen
lefarcen requested a review from nettee July 21, 2026 10:20
Siri-Ray added 5 commits July 22, 2026 15:34
Generated-By: looper 0.11.0 (runner=fixer, agent=codex)
Generated-By: looper 0.11.0 (runner=fixer, agent=codex)
Make the conversation-message boundary explicit so the test observes the loading-disabled state before asserting that question-form submission becomes available.

Generated-By: looper 0.11.0 (runner=fixer, agent=codex)
Generated-By: looper 0.11.0 (runner=fixer, agent=codex)
Generated-By: looper 0.11.0 (runner=fixer, agent=codex)
@Siri-Ray
Siri-Ray added this pull request to the merge queue Jul 28, 2026
@Siri-Ray
Siri-Ray removed this pull request from the merge queue due to a manual request Jul 28, 2026
Comment thread apps/daemon/tests/opencode-session-resume.test.ts

@AmyShang-alt AmyShang-alt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QA Request changes:本地在 PR head 336125d 上验证,daemon context-budget 相关单测 52/52、contracts 5/5、typecheck 通过;但 opencode-session-resume.test.ts 的 rollover 集成用例连续两次失败。第二轮 rollover run 在 10 秒内未进入 succeededfailedcanceled,报 run ... did not finish。这是本 PR 的核心 runtime 路径,当前不能接受合并。请修复 rollover 后 run 不结束的问题,并补充稳定通过的 runtime 验证。

Resolve the run-finished analytics contract by preserving both context-budget and current AMR/OpenCode diagnostics. Give the real daemon/CLI rollover integration a failure budget that does not misclassify slow machines, and retain the last observed run details on timeout.

Generated-By: looper 0.11.1 (runner=fixer, agent=codex)
@Siri-Ray

Copy link
Copy Markdown
Contributor Author

Looper fixer round completea81037b

  • ✅ Review comment on apps/daemon/tests/opencode-session-resume.test.ts:202 (@AmyShang-alt) — thread
    • Updated apps/daemon/tests/opencode-session-resume.test.ts with a realistic 30-second runtime completion budget and actionable timeout diagnostics. The rollover case passed three consecutive focused runs and the full file passed 6/6.

🔁 Powered by Looper · runner=fixer · agent=codex · An autonomous AI dev team for your GitHub repos.

Siri-Ray added 8 commits July 29, 2026 17:42
Generated-By: looper 0.11.1 (runner=fixer, agent=codex)
Generated-By: looper 0.11.1 (runner=fixer, agent=codex)
Generated-By: looper 0.11.8 (runner=fixer, agent=codex)
Generated-By: looper 0.11.8 (runner=fixer, agent=codex)
Generated-By: looper 0.11.8 (runner=fixer, agent=codex)
Preserve resume-state instruction rebuilding while adopting the mainline removal of the Codex image-generation prompt override. Recalibrate the low-context rollover fixture for the current stable-prompt size.

Generated-By: looper 0.11.8 (runner=fixer, agent=codex)
Preserve effective input-token analytics from the context-budget branch while adopting mainline batched append-only message event persistence.

Generated-By: looper 0.11.8 (runner=fixer, agent=codex)

@AmyShang-alt AmyShang-alt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Siri-Ray

Thanks for the follow-up. I re-ran the rollover validation on the current head and found one remaining test-harness blocker. Please align the outer Vitest timeout with the run-completion polling budget, then re-run the full file with the default repository command.

Comment thread apps/daemon/tests/opencode-session-resume.test.ts
Generated-By: looper 0.11.8 (runner=fixer, agent=codex)
@Siri-Ray

Copy link
Copy Markdown
Contributor Author

Looper fixer round complete4583df9

  • ✅ Review comment on apps/daemon/tests/opencode-session-resume.test.ts:47 (@AmyShang-alt) — thread
    • Added a scoped 60-second timeout to all three rollover integration tests in apps/daemon/tests/opencode-session-resume.test.ts, keeping the outer Vitest budget above the 30-second polling window. The default full-file command passes 6/6 without a CLI timeout override.

🔁 Powered by Looper · runner=fixer · agent=codex · An autonomous AI dev team for your GitHub repos.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-validation Runtime change detected; needs human or /explore agent validation. risk/high High risk: apps/desktop, daemon, auth, migration, workflows, package deps size/XL PR changes 700-1500 lines type/bugfix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants