Skip to content

Commit abdcbb8

Browse files
endolinbotclaude
andcommitted
mentor: classify an empty-output handler kill (rc=137/143/offline) as transient, not a die
The improve-handler branch classified a failure transient only when the captured output matched is_transient_claude_signature or _fetch_stderr_is_offline. An empty-output non-zero exit — a `claude -p` handler SIGKILLed/OOM-killed (rc=137/143) or cut mid-call by a quota/network blip — produces an EMPTY $capture that matches neither TEXT signature, so it fell through to die (exit 1). That marks garden-mentor.service Failed and fires self-heal-run.sh into a diagnosis that fails identically in the same outage — the observed 30-min FATAL loop on endolinbot (2026-07-05 18:20/18:50/19:20, bare "FATAL: improve handler failed" with no preceding $out diagnostic). Capture the handler's exit code immediately into $rc (under set +e, mirroring gardener.sh:376-381) so it survives the later out="$(...)" substitution, and add an empty-capture-transient branch mirroring gardener.sh:679-687: when the capture is empty and is_transient_empty_failure "$rc" is true, WARN + exit 0 (markers unadvanced, retry next tick) rather than die. Reuses the same common.sh helper the gardener relies on, keeping the two handlers' transient-vs-real classification aligned. Reserves the die->self-heal path for a non-empty or non-transient real defect. Adds a regression subtest (rc=137 empty-output kill -> exit 0, markers unadvanced) alongside the existing transient/real cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 2e0ea48 commit abdcbb8

2 files changed

Lines changed: 43 additions & 1 deletion

File tree

scripts/jobs/mentor.sh

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,10 +96,36 @@ rm -f "$digest" # the blob now lives in $DIR's object DB; the temp file is spe
9696
# the die (exit 1 → self-heal) path for a genuine, non-transient handler
9797
# failure. Mirrors gardener.sh's transient-vs-real classification.
9898
capture="$(mktemp "${TMPDIR:-/tmp}/garden-mentor-capture.XXXXXX")"
99-
if "$GARDEN_MENTOR_HANDLER" "$sha" "$DIR" >"$capture" 2>&1; then
99+
# Capture the handler's exit code IMMEDIATELY into $rc. A bare `if
100+
# "$HANDLER" …; then` would work for the success arm, but the failure arm below
101+
# needs $rc to classify an EMPTY-output kill (rc=137/143 SIGKILL/OOM/SIGTERM, or
102+
# the offline rc) — and `$?` there would already be clobbered by the `out="$(…)"`
103+
# substitution. Run the handler as a plain statement under `set +e` (mirroring
104+
# gardener.sh:376-381) so a non-zero exit is captured into $rc rather than
105+
# tripping `set -e` before the classifier below runs.
106+
set +e
107+
"$GARDEN_MENTOR_HANDLER" "$sha" "$DIR" >"$capture" 2>&1
108+
rc=$?
109+
set -e
110+
if [ "$rc" -eq 0 ]; then
100111
for f in "${new[@]}"; do printf '%s\n' "${f#"$DIR"/}" >> "$SEEN"; done
101112
date -u +%FT%TZ > "$JSINCE"
102113
rm -f "$capture"
114+
elif [ ! -s "$capture" ] && is_transient_empty_failure "$rc"; then
115+
# EMPTY capture + a signal/offline rc — a `claude -p` handler SIGKILLed/
116+
# OOM-killed (rc=137/143) or cut mid-call by a quota/network blip that flushed
117+
# nothing to $capture — matches neither is_transient_claude_signature nor
118+
# _fetch_stderr_is_offline (both need signature TEXT), so without this branch it
119+
# would fall through to `die` and fire self-heal-run.sh into a diagnosis that
120+
# fails identically in the SAME outage (the observed 30-min FATAL loop on
121+
# endolinbot, 2026-07-05 18:20/18:50/19:20, bare "FATAL: improve handler failed"
122+
# with no preceding $out diagnostic → empty output). Classify it transient,
123+
# mirroring gardener.sh's empty-capture branch (gardener.sh:679-687) via the
124+
# SAME common.sh helper so the two handlers stay aligned: WARN + exit 0, leaving
125+
# $SEEN/$JSINCE unadvanced so the next tick retries.
126+
rm -f "$capture"
127+
log "WARN: improve handler killed with empty output (rc=$rc); leaving markers, retrying next tick"
128+
exit 0
103129
else
104130
out="$(tail -c 65536 "$capture" 2>/dev/null || true)"
105131
rm -f "$capture"

scripts/jobs/test/run-test.sh

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -916,6 +916,22 @@ seen_after="$(wc -l < "$SEENF" 2>/dev/null || echo 0)"
916916
rrc=0; env GARDEN_MENTOR_HANDLER="$TR/mentor-real.sh" "$JOBS/mentor.sh" >/dev/null 2>&1 || rrc=$?
917917
[ "$rrc" -ne 0 ] && ok "genuine handler defect → non-zero exit (self-heal path)" \
918918
|| bad "real defect did not die (rc=$rrc)"
919+
# An EMPTY-output SIGKILL/OOM (rc=137) — a `claude -p` handler killed mid-call
920+
# leaving nothing in $capture — matches no transient SIGNATURE (the capture is
921+
# empty), so before the empty-capture-transient branch it fell through to `die`
922+
# and fired self-heal into the same outage (the 2026-07-05 30-min FATAL loop on
923+
# endolinbot). It must be absorbed: WARN + exit 0, markers unadvanced.
924+
cat > "$TR/mentor-emptykill.sh" <<'EOF'
925+
#!/bin/bash
926+
exit 137
927+
EOF
928+
chmod +x "$TR/mentor-emptykill.sh"
929+
seen_before_k="$(wc -l < "$SEENF" 2>/dev/null || echo 0)"
930+
krc=0; env GARDEN_MENTOR_HANDLER="$TR/mentor-emptykill.sh" "$JOBS/mentor.sh" >/dev/null 2>&1 || krc=$?
931+
seen_after_k="$(wc -l < "$SEENF" 2>/dev/null || echo 0)"
932+
{ [ "$krc" -eq 0 ] && [ "$seen_after_k" -eq "$seen_before_k" ]; } \
933+
&& ok "empty-output kill (rc=137) → exit 0, markers unadvanced (retry next tick)" \
934+
|| bad "empty-output kill not absorbed (rc=$krc seen $seen_before_k$seen_after_k)"
919935

920936
# ============================================================================
921937
hr; echo "SUBTEST 12 — CURSORS: durable poll position survives a restart"; hr

0 commit comments

Comments
 (0)