Unify streaming retry, recovery, and failover under one committed request budget #2532

Objective

Replace LLxprt's independently composed retry/recovery paths with one request-scoped retry architecture that preserves transient recovery, Anthropic HTTP-200 overload handling, OAuth repair, credential-bucket rotation, and load-balanced failover while making replay after observable stream output impossible.

This is not a request to remove retries. Distinct recovery decisions must share one failure taxonomy, aggregate budget, and irreversible commit state.

Problem and file catalog

• packages/providers/src/RetryOrchestrator.ts: central retry/backoff, first-chunk timeout, auth recovery, bucket rotation. yieldStreamUnprotected() marks post-yield failures, but streamWithTimeout()/raceFirstChunkWithTimeout() use a separate path.
• packages/providers/src/anthropic/AnthropicStreamProcessor.ts: parses events and also retries networks in processAnthropicStream(). Usage metadata does not consistently set hasYieldedContent.
• packages/providers/src/anthropic/AnthropicProvider.ts
• packages/providers/src/LoadBalancingProvider.ts: guards post-yield failover for immediate classes such as 429, but not universally for ordinary retryable failures.
• packages/providers/src/loadBalancing/streamTimeout.ts: separate timeout/iterator cleanup.
• packages/providers/src/ProviderManager.ts
• error/auth/bucket helpers under packages/providers/src/**
• active retry helpers under packages/core/src/utils/retry.ts
• packages/providers/src/RetryOrchestrator*.test.ts
• packages/providers/src/LoadBalancingProvider*.test.ts
• Anthropic tests under packages/providers/src/anthropic/**

Anthropic can return HTTP 200 and later send SSE event: error, including overloaded_error. This remains retryable before any observable event, but never after metadata, text, thinking, or a tool call escapes.

Known gaps:

1. streamWithTimeout() loses normal post-yield marking after the first chunk.
2. First-chunk timeout does not consistently close the losing iterator/request.
3. Anthropic owns a duplicate retry loop and can retry after usage metadata.
4. Load balancing does not universally block retry/failover after output.
5. Distributed counters and delays allow multiplicative attempts.
6. Provider-terminal validation for malformed/truncated streams is not unified.

Required architecture

Represent at least:

phase: connect | headers | stream | protocol | auth | tool | cancellation
kind: timeout | network | rate_limit | overload | server | auth | payment |
      malformed | truncated | invalid_request | cancelled | unknown
status?: number
retryAfterMs?: number
providerCode?: string
exposure: none | metadata | content | tool_call
terminalSeen: boolean
cause: unknown

Track aggregate attempts, deadline, total wait, visited credentials/targets, Retry-After, and an irreversible committed flag set before every outward event.

Mandatory policy:

• After exposure, do not replay, rotate credentials for this request, retry a backend, or fail over.
• Metadata counts as exposure.
• HTTP-200 in-band errors remain retryable before exposure.
• Auth repair after exposure can prepare future requests but cannot replay this one.
• Timeout owns and closes the losing iterator.
• Adapters decode provider errors; shared policy decides recovery.
• Preserve tool pairing/deduplication and history integrity.
• Configure/document unavoidable SDK retries against the aggregate budget.

Required characterization tests

• Anthropic usage metadata then reset
• HTTP-200 overload before output, after metadata, and after text
• first-chunk timeout retries before output and closes losing iterator
• timeout-enabled partial text then network, 429, 5xx, overload: no replay
• load-balanced partial text then ordinary network/5xx/aggregate failure: no retry/second backend
• partial thinking/tool assembly then failure
• malformed event and EOF without terminal event
• cancellation during first-chunk race
• bounded aggregate attempts across SDK/provider/auth/bucket/load-balancer layers

Acceptance criteria

• One documented taxonomy and request budget governs central retry, auth recovery, bucket rotation, and load-balanced target decisions.
• One guarded-stream primitive handles exposure and cleanup for normal and timeout streams.
• Commitment is set before metadata, text, thinking, or tool output.
• No replay, credential rotation, or target failover occurs after commitment.
• Anthropic HTTP-200 overload/rate-limit/wrapped errors still recover before exposure.
• Retry-After is normalized within the shared budget.
• Anthropic no longer owns an unbudgeted network replay loop.
• Losing timeout iterators are cancelled/closed.
• Malformed/truncated streams are not committed as successful turns.
• OAuth, cross-process token recovery, buckets, load balancing, cancellation, history, tools, and headless mode retain functionality.
• Tests prove partial output is not duplicated or mixed.
• Retry telemetry reports phase, kind, budget, target, and commitment without secrets.
• Architecture docs assign decoding and recovery ownership.

Verification

Run npm run format, npm run lint, npm run typecheck, npm run test, npm run build, and bun scripts/start.ts --profile-load ollamakimi "write me a haiku and nothing else", plus focused retry/Anthropic/OAuth/bucket/load-balancer/cancellation/noninteractive tests.

Appreciable outcome

LLxprt still recovers from failures that motivated its retry system—including Anthropic's non-HTTP overloads—but each request has a predictable budget, no leaked timeout request, no duplicate/mixed output after exposure, and one maintainable recovery model.