Fix the top remaining failed-run bucket after PR1 and retry data

[yinjialu]

Parent: #3408
Spec: #3526
Depends on: #3543, #3544

Problem

After PR #3412 and the safe retry slice, remaining failures should be fixed by measured volume and fixability rather than intuition. This issue tracks the first ROI-ranked failure fix selected from PostHog run_finished data.

Scope

• Wait for enough post-PR1 / post-retry data to rank failure buckets.
• Select one top bucket using failure_category, failure_detail, failure_stage, agent_provider_id, model_id, and retry outcome.
• Inspect representative Langfuse traces via langfuse_trace_id where available.
• Land one focused fix with a red/fixed validation path.
• Document adjacent buckets that are intentionally left for follow-up.

Acceptance criteria

• The selected bucket is justified with PostHog volume and impact data.
• Representative traces or logs explain the actual failure mode.
• The implementation reduces the selected bucket without increasing adjacent failure buckets.
• The fix preserves the run terminal analytics invariants from PR Improve run failure classification and Langfuse correlation #3412.

Validation

• Before/after PostHog query or dashboard screenshot/link for the selected bucket.
• Focused test or replay that fails before the fix and passes after it.
• Fixed-task QA evidence once Create fixed-task QA gate for run reliability optimizations #3545 is available.

Data needed before implementation

• PostHog breakdown of failed run_finished by failure fields and provider/model.
• Retry outcome data if Implement safe retry policy and retry telemetry for chat runs #3543 has landed before this work starts.

[lefarcen]

@yinjialu The sequencing here makes sense: wait for the post-#3412 / post-retry sample to settle, then pick the highest-volume bucket with trace evidence instead of guessing.

Tagged this as a bug-tracking item. When the first fix lands, the main thing we should preserve is the run terminal analytics invariant from #3412 while showing a before/after drop for the selected failure bucket.

[yinjialu]

Data-driven bucket selection for #3546

Fresh 0.10.0 release-health read, rolling 7 days:

• Total run_finished: 9,310.
• Non-success: 2,733 / 29.4%.
• Actionable non-success excluding user_cancel: 2,123 / 22.8% of all runs.

The first ROI-ranked failure fix should target AMR/account-state prevention rather than expanding retry:

• auth / auth_required: 594.
• insufficient_balance / amr_insufficient_balance: 565.
• Combined account-state failures: ~1,159 events, about 42% of non-success and 12% of all 0.10.0 runs.

Top provider/model contributors:

• amr / deepseek-v4-flash: 521 total, 320 non-success, 311 actionable failures, 59.7% actionable failure rate.
• amr / default: 220 total, 211 non-success, 205 actionable failures, 93.2% actionable failure rate.
• amr / claude-opus-4.8: 198 total, 174 non-success, 167 actionable failures, 84.3% actionable failure rate.
• amr / gpt-5.5: 81 total, 64 non-success, 61 actionable failures, 75.3% actionable failure rate.

Representative Langfuse traces confirm these are account-state failures, not transient runtime failures:

• 287588ba-4f44-47a0-af7e-2435371196d4: AMR deepseek-v4-flash, insufficient balance. Langfuse status says to recharge the AMR wallet.
• 0f90cc9a-fb57-4526-828c-2b1e6afed11c: AMR claude-opus-4.8, auth required. Langfuse status says AMR sign-in is required.

Recommended implementation target:

• Add AMR account-state preflight/gating before a run becomes a terminal agent failure.
• For AMR defaults/model choices, detect auth/credit availability early and route to login/recharge guidance instead of launching a doomed run.
• Keep the terminal analytics invariant: if a run still reaches failure, it must preserve error_code, failure_category, failure_stage, retryable, and user_action.

This should be the #3546 first fix unless a newer dashboard read shows the AMR account-state bucket dropping below process/timeout buckets.

[lefarcen]

@yinjialu This 7-day read makes the ranking pretty clear — the AMR account-state bucket is a better first target than retry expansion.

The strongest signal to me is the concentration in auth_required + amr_insufficient_balance, plus the amr / default failure rate staying so high. That points to preventing doomed runs earlier rather than spending another slice on post-launch recovery. The Langfuse traces you pulled also make the implementation seam concrete: preflight auth/credit gating for AMR-backed defaults and model picks, while keeping the run_finished terminal fields stable if a run still reaches failure.

Let's use this as the implementation target for #3546. When the PR goes up, please link the PostHog snapshot / query and the two representative trace ids in the description so the before/after verification stays attached to the fix.

[lefarcen]

@yinjialu Agree with locking the first fix around AMR account-state prevention.

The extra thing I'd push into the implementation scope is making the preflight decision explicit for both the default AMR path and an explicit AMR model pick, so we don't only fix one entry path and leave the other still failing terminally. If a newer dashboard cut doesn't displace this bucket, this comment + the trace IDs above are enough to treat #3546 as the selected slice.

When the PR opens, please call out which user-action branch each preflight maps to (login vs recharge) alongside the preserved terminal fields — that will make the before/after verification much easier to read.

[lefarcen]

Quick tracker link: draft PR #4455 is now open for the AMR Cloud Recovery vertical slice that turns AMR insufficient-balance/account-recovery states into a recovery overlay instead of generic failed-run behavior.

That looks directly relevant to this issue's AMR account-state prevention lane because it adds the daemon recovery service, run/API recovery surfaces, and the web recovery affordances around continue/cancel/restart. Keeping this issue open until the implementation path lands cleanly.

[github-actions]

This issue has been inactive for 30 days and is being marked as stale. If it is still relevant, please leave a comment or push an update — otherwise it will be closed in 7 days.

This is only a queue-management step, not a rejection of the report. Closed issues can be reopened when work resumes.

[lefarcen]

@yinjialu Your June 12 bucket read still looks like the right framing for this tracker: #3412 landed the classification/correlation foundation, and the AMR account-state prevention lane was the next logical slice from the data you pulled.

What changed since then is implementation state rather than direction: the AMR recovery attempt later moved into PR #4455, but that PR was closed without merging, so this issue is still the right place to keep the account-state bucket open until a fresh implementation slice lands.

I’m clearing the stale marker here so the tracker stays open.