mentor-claude: propagate transient inner-claude failure instead of swallowing it

The `claude -p` call ran under `set -euo pipefail`, so a non-zero exit — the
common transient case (quota/usage cut, Anthropic overload/5xx, api/network
blip) — aborted the handler at rc=1 with claude's stdout captured into a
discarded $out and nothing re-emitted. mentor.sh then saw an empty capture +
rc=1, which matches neither is_transient_empty_failure (wants a signal/offline
rc) nor is_transient_claude_signature/_fetch_stderr_is_offline (both need
signature TEXT), so it fell through to `die` and misread a transient inner-agent
outage as a real handler defect — needlessly marking garden-mentor.service
Failed and firing self-heal-run.sh into an identically-failing diagnosis (the
30-min FATAL loop observed 2026-07-06 00:50/01:20/01:50).

Mirror gardener-claude.sh / follow-up-claude.sh: run the inner agent under
`set +e`, capture rc and combined stdout+stderr, restore `set -e`, and on
non-zero rc re-emit the captured output to stderr AND propagate rc via `exit`
so mentor.sh's transient classifiers can see the signature text and/or a
propagated signal/offline rc and correctly WARN-and-retry. The rc=0 path parses
$out for JOB blocks exactly as before. Route the call through a
GARDEN_MENTOR_CLAUDE seam (default `claude`) mirroring GARDEN_FOLLOWUP_CLAUDE so
the transient-propagation path is testable with a non-`claude`-named stub (the
fleet sandbox pins the literal `claude` binary). Add a run-test.sh regression
driving the real handler through fake-claude.sh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: endolinbot

scripts/jobs/handlers/mentor-claude.sh

@@ -57,12 +57,40 @@ below. Emit nothing if there is no clear opportunity.
 EOF
 )"
 
-command -v claude >/dev/null 2>&1 || die "claude not on PATH; cannot run mentor"
+# The inner agent is `claude` by default; tests inject a deterministic stub via
+# GARDEN_MENTOR_CLAUDE so the parse/post path and the transient-propagation path
+# below can be driven without a live model (mirrors GARDEN_FOLLOWUP_CLAUDE in
+# follow-up-claude.sh — and a non-`claude`-named stub is required because the
+# fleet sandbox pins the literal `claude` binary, so a PATH shim named `claude`
+# cannot execute).
+: "${GARDEN_MENTOR_CLAUDE:=claude}"
+command -v "$GARDEN_MENTOR_CLAUDE" >/dev/null 2>&1 \
+  || die "$GARDEN_MENTOR_CLAUDE not on PATH; cannot run mentor"
 # --dangerously-skip-permissions: autonomous headless context, no human
 # approver; the default permission gate would deny every tool call. Bypass is
 # the intended fleet posture (operator pre-consents via
 # skipDangerousModePermissionPrompt in ~/.claude). Requires running as non-root.
-out="$(claude -p --dangerously-skip-permissions "$prompt")"
+# Run `claude` under `set +e` so a NON-ZERO exit — the common transient case (a
+# quota/usage cut, an Anthropic overload/5xx, an api/network blip) — is captured
+# rather than tripping `set -e`, which would abort this handler at rc=1 with
+# claude's output trapped in a discarded `$out` and NOTHING re-emitted. mentor.sh
+# captures our combined stdout+stderr and classifies it: with empty output + rc=1
+# it matches neither is_transient_empty_failure (wants a signal/offline rc) nor
+# is_transient_claude_signature/_fetch_stderr_is_offline (both need signature
+# TEXT), so it falls through to `die` and misreads a transient inner-agent outage
+# as a real handler defect — the 30-min FATAL loop observed 2026-07-06. Mirror
+# gardener-claude.sh:279-282: on non-zero rc, re-emit the captured output to our
+# stderr (so mentor.sh's classifiers can see the transient signature and/or a
+# propagated signal/offline rc) AND propagate rc via `exit`. The rc=0 path parses
+# `$out` for JOB blocks exactly as before.
+set +e
+out="$("$GARDEN_MENTOR_CLAUDE" -p --dangerously-skip-permissions "$prompt" 2>&1)"
+rc=$?
+set -e
+if [ "$rc" -ne 0 ]; then
+  printf '%s\n' "$out" >&2
+  exit "$rc"
+fi
 
 # already_fixed_pending_deploy: deterministic pre-filter (no LLM). A mentor
 # "improve <script>" job is churn if the fix is ALREADY committed to

scripts/jobs/test/run-test.sh

@@ -933,6 +933,26 @@ seen_after_k="$(wc -l < "$SEENF" 2>/dev/null || echo 0)"
   && ok "empty-output kill (rc=137) → exit 0, markers unadvanced (retry next tick)" \
   || bad "empty-output kill not absorbed (rc=$krc seen $seen_before_k→$seen_after_k)"
 
+# The REAL mentor-claude.sh handler must PROPAGATE a transient inner-`claude`
+# failure (a quota/usage cut, an Anthropic overload/5xx, an api/network blip) — a
+# non-zero exit with its output RE-EMITTED — not swallow it. Before the fix the
+# `claude` call ran under `set -euo pipefail`, so a non-zero exit aborted the
+# handler at rc=1 with claude's stdout trapped in a discarded `$out` and NOTHING
+# re-emitted; mentor.sh then saw an empty capture + rc=1, matched no transient
+# classifier, and fell through to `die`/self-heal — the 2026-07-06 FATAL loop.
+# Drive the real handler with the deterministic fake-claude stub (via the
+# GARDEN_MENTOR_CLAUDE seam — a differently-named binary because the fleet sandbox
+# pins the literal `claude`, so a PATH shim named `claude` cannot execute) failing
+# with a transient signature; the handler must re-emit that signature AND exit
+# non-zero so mentor.sh's classifier can absorb it.
+mc_rc=0
+mc_out="$(env GARDEN_MENTOR_CLAUDE="$HERE/fake-claude.sh" FAKE_CLAUDE_FAIL=1 \
+  FAKE_CLAUDE_STDERR="API Error: Overloaded (529) — please retry" \
+  GARDEN_ROOT="$JOBS/../.." "$JOBS/handlers/mentor-claude.sh" deadbeef "$TR" 2>&1)" || mc_rc=$?
+{ [ "$mc_rc" -ne 0 ] && printf '%s' "$mc_out" | grep -q 'Overloaded (529)'; } \
+  && ok "mentor-claude propagates transient claude failure (rc≠0, output re-emitted)" \
+  || bad "mentor-claude swallowed transient claude failure (rc=$mc_rc out='$mc_out')"
+
 # ============================================================================
 hr; echo "SUBTEST 12 — CURSORS: durable poll position survives a restart"; hr
 export GARDEN=curhost