Skip to content

fix(loop): retry recoverable stream/parse error instead of killing the run#74

Merged
arniwesth merged 1 commit into
arniwesth:mainfrom
sunholo-voight-kampff:fix/stream-error-retry
Jun 27, 2026
Merged

fix(loop): retry recoverable stream/parse error instead of killing the run#74
arniwesth merged 1 commit into
arniwesth:mainfrom
sunholo-voight-kampff:fix/stream-error-retry

Conversation

@sunholo-voight-kampff

Copy link
Copy Markdown
Contributor

Problem

A stream/parse error mid-step — e.g. the AILANG std/xml provider raising "unexpected EOF" while parsing a partial generation — currently terminates the whole run (finish=error). One transient hiccup kills an otherwise-healthy session. This is the failure mode behind several "the model died" reports that turned out to be harness-side, not model-side.

Fix (+16 / −2, agent_loop_v2.ail)

Treat a recoverable stream/parse error the same way #13 (truncation) and #19 (empty response) are already treated: re-issue the step (fresh generation) instead of bailing.

  • Bounded by the existing step_budget — can't loop forever.
  • Gated by MOTOKO_RETRY_STREAM_ERROR (default 1); set 0 to restore the old kill-on-error behavior.
  • Emits a stream_error_retry telemetry event so it's visible, not silent.

Caveat (why this is draft)

This is a backstop, not a root-cause fix. I haven't yet pinned the exact source of the std/xml "unexpected EOF" — that needs a debug build to capture the raw partial generation. The retry stops it from killing runs today, but you may prefer:

  • a more targeted retry (only the specific xml-parse error class, not any recoverable error), and/or
  • fixing the parser root cause first and keeping the loop strict.

Posting as draft to decide scope. The env gate + telemetry event make it safe to ship as-is if you'd rather have the backstop now and chase the root cause separately — your call.

🤖 Generated with Claude Code

… the run

dispatch_step Err (e.g. a Go-side 'XML syntax error ... unexpected EOF' from
AILANG std/xml choking on the model's malformed DOCX XML, surfaced as AIError
code=Internal) terminated the whole run with finish=error at the step it
happened. Now re-issue the step (fresh generation), bounded by step_budget.
Same pattern as arniwesth#13 (length) / arniwesth#19 (empty). Disable: MOTOKO_RETRY_STREAM_ERROR=0.
@sunholo-voight-kampff sunholo-voight-kampff marked this pull request as ready for review June 26, 2026 16:51

@arniwesth arniwesth left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arniwesth arniwesth merged commit 154acb4 into arniwesth:main Jun 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants