feat(infra): implement blue-green deployment pipeline — parallel environment promotion, automated smoke test gate, feature-flag-controlled traffic shift, and one-command rollback

[Chibey-max]

Overview

Zero-downtime deployments are difficult when both the backend and Soroban contracts must be updated atomically. A blue-green deployment pipeline maintains two production environments (blue = current, green = next), shifts traffic via a feature flag, runs automated smoke tests on green before committing, and rolls back with a single command if smoke tests fail.

What to build

• GitHub Actions workflow blue-green-deploy.yml triggered on push to main:
  1. Build and push Docker image to registry, tagged :{sha}
  2. Deploy to green environment (ECS/K8s replica set); green runs alongside blue on a separate port
  3. Run smoke test suite (Playwright + k6 single-user test against green's health endpoints)
  4. If smoke tests pass: update the load balancer to shift 100% traffic to green; tag previous blue for teardown after 10-min drain
  5. If smoke tests fail: tear down green; notify via Slack webhook; create a GitHub issue automatically
• Smoke test suite (tests/smoke/):
  • GET /healthz returns 200
  • POST /api/v1/auth/login with a test user succeeds
  • GET /api/v1/escrows returns a 200 with the expected schema
  • Soroban RPC connectivity check: GET /api/v1/contracts/status returns { connected: true }
• Feature flag gate: wrap the traffic shift in a check for the enable_blue_green_cutover feature flag (feat(backend): build per-tenant feature flag service — flag definitions with rollout percentages, user-segment overrides, kill-switch support, and REST management API #1528); if the flag is off, pipeline stops after green deployment and waits for manual approval
• Rollback script (scripts/rollback.sh): shifts traffic back to blue, tears down green, sends a Slack alert
• DEPLOYMENT.md: documents the blue-green process, manual override steps, and rollback procedure

Acceptance Criteria

• Blue environment receives 0 interruption during green deployment and smoke test phase
• Traffic shift is atomic at the load balancer level (no partial split during cutover)
• Smoke test failure triggers rollback automatically within 2 minutes of failure detection
• A GitHub issue is created on smoke test failure with: commit SHA, environment, failed test names, and logs URL
• Feature flag check is a hard gate — pipeline exits 0 (not an error) and leaves green running when flag is off
• Rollback script is idempotent — running it twice when already on blue is a no-op
• DEPLOYMENT.md covers: normal flow, flag-gated cutover, manual rollback, drain period

Technical notes

• Load balancer abstraction: use AWS ALB target group weights or an nginx upstream weight; abstract behind a scripts/shift-traffic.sh {blue|green|split:N} helper
• Green environment URL is deterministic: green.{base_domain} configured via env var

[Cmitchelle7]

Hi Maintainer, reviewed the issue in detail. Full stack background with solid experience relevant to this task. No blockers on my end — I'm ready to submit a PR shortly. Please assign.

[grantfox-oss]

🦊 GrantFox — @Cmitchelle7 has been assigned to this issue!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #1544)
2. Your PR will be reviewed by the Stellar-Trust-Escrow maintainers

Good luck! Track your progress on GrantFox.