Skip to content

fix: harden E2E pipeline — phase detection, log-based health checks, image_tag bypass - #3

Merged
vhrahulkumar merged 7 commits into
mainfrom
feat/upgrade-automation
Mar 23, 2026
Merged

fix: harden E2E pipeline — phase detection, log-based health checks, image_tag bypass#3
vhrahulkumar merged 7 commits into
mainfrom
feat/upgrade-automation

Conversation

@vhrahulkumar

Copy link
Copy Markdown
Owner

Summary

Hardens the automated OpenClaw upgrade E2E pipeline. All 7 health checks passed on Run digitalocean-labs#12.

Root causes fixed

# Symptom Root cause Fix
1 All health checks failed with unknown flag: --command doctl apps console has no --command flag — interactive only Replaced with runtime log polling via doctl apps logs --type=run
2 Deployment stuck at UNKNOWN for 600s Phase detection only checked active_deployment.phase; new apps track in-progress work in pending_deployment Check both; treat ERROR/CANCELED in either as immediate failure with log dump
3 Exit code 5 on APP_ID extraction jq 1.7+ exits 5 when // empty produces no output; set -euo pipefail kills the script Replace // empty with // "" + add `
4 Exit code 1 — APP_ID always empty doctl apps create 2>&1 prepends Notice: App created to stdout before the JSON, breaking jq Strip Notice: lines before piping to jq; silent grep fallback
5 Exit code 5 in phase poll loop doctl apps get returns [{app}] (array); .pending_deployment.phase on an array causes jq type error → exit 5 Normalise with (if type == "array" then .[0] else . end) + `

Changes

tests/e2e/deploy-and-test.sh

  • Phase detection polls both pending_deployment.phase and active_deployment.phase, handles array response from doctl apps get
  • Health checks use doctl apps logs --type=run log pattern matching instead of doctl apps console --command
  • Robust APP_ID extraction: strip Notice: prefix, jq with // "", silent grep fallback
  • All jq calls guarded with || true

.github/workflows/openclaw-upgrade-check.yml

  • Added image_tag workflow_dispatch input — provide an existing GHCR tag to skip Docker build/push and go straight to E2E
  • docker-tests and push-e2e-image skip when image_tag is provided
  • open-pr and open-issue skip in bypass mode

Test results (Run digitalocean-labs#12)

[e2e] Deployment active after 120s
[e2e] Gateway startup detected (~30s)
[e2e] Running E2E health checks...
  ✓ Config generated
  ✓ tools.profile: coding
  ✓ Gateway process started
  ✓ Auth mode: token
  ✓ Channel plugins present
  ✓ Feature flags applied (tailscale/ngrok/ssh disabled)
  ✓ All init scripts exited 0

[e2e] E2E Results: 7 passed, 0 failed
[e2e] All E2E checks passed ✅

jq 1.7+ exits with code 5 when a top-level expression produces no
output (e.g. when // empty is triggered). Under set -euo pipefail this
kills the script silently. Replace with // "" so jq always produces
output and exits 0, then check for empty string as before.
jq fails when doctl stderr output (2>&1) prefixes the JSON with a
warning. Use the grep pattern that worked in earlier runs as the
fallback. Also log the first line of doctl output so we can see any
non-JSON prefix that was tripping up jq.
doctl apps get returns [{app}] (array), not a plain object. Calling
.pending_deployment.phase directly on an array causes a jq type error
which exits with code 5 under set -euo pipefail, killing the loop
before the first Phase: line is printed. Normalise to object with
(if type == "array" then .[0] else . end) and add || true guards.
doctl outputs 'Notice: App created' to stdout before the JSON.
Strip all Notice: lines so jq parses the JSON directly and the
grep fallback is no longer needed.
Remove spec dump, doctl exit code log, first-line/last-20-lines
output dumps, and all DEBUG: log messages added during testing.
Keep only production-relevant log output and error handling.
@vhrahulkumar
vhrahulkumar merged commit c4b4438 into main Mar 23, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant