migration-test: the report still says "Result: PASSED" when the failure is in one of the 4 steps between phase_latest and phase_nightly_api

[rafaelgiln]

Follow-up to #1120 (PR #1139). That work stopped the report inferring success from silence for the three verification phases: the runner's steps.<id>.outcome is now handed to generate_report.py as PHASE_OUTCOME_<phase> and reconciled against what the phase recorded, so a phase that crashed before writing its verdict is reported as unaccounted instead of rendering as Result: PASSED.

Only three steps carry an id, so only three steps are reconciled. The steps between them are not, and a failure there reproduces the original #1120 symptom by a different route: an issue titled "Langflow Migration Test Failed" whose body opens with Result: PASSED.

What is wrong

migration-test.yml has four steps that can fail after phase_latest succeeded and before phase_nightly_api runs:

Line	Step	How it fails
migration-test.yml:146	Resolve nightly version from PyPI	explicit exit 1 when PyPI returns no pre-release
migration-test.yml:167	Install Langflow nightly	uv resolution failure
migration-test.yml:181	Start Langflow nightly (same database)	process dies immediately
migration-test.yml:190	Wait for Langflow nightly (includes migration)	180 s timeout — the alembic migration failing to complete is exactly what this workflow exists to catch

When any of them fails the job stops. Verify migration via API and Verify migration via UI never run, so steps.phase_nightly_api.outcome and steps.phase_nightly_ui.outcome evaluate to the empty string; declared_outcomes() filters empty values, so neither phase is declared. The state file holds only the latest phase, all steps pass. assess() finds no failure, no integrity problem, and returns PASSED.

Generate report runs under if: always(), and Create or update issue on failure runs under if: failure() — so both fire, and the issue body contradicts its own title.

The most valuable failure this workflow can detect — the nightly not booting against a migrated database — lands squarely in this gap.

Reproduction

No CI needed; the script is driven entirely by STATE_FILE + the PHASE_OUTCOME_* variables. This is the exact shape of a run whose nightly never came up:

cat > /tmp/state.json <<'JSON'
{"flow_name":"witness","flow_id":"1",
 "phases":{"latest":{"steps":{"create":{"status":"pass"},"execute":{"status":"pass"}}}}}
JSON

env -i PATH="$PATH" STATE_FILE=/tmp/state.json REPORT_FILE=/tmp/report.md \
  PHASE_OUTCOME_latest=success PHASE_OUTCOME_nightly_api= PHASE_OUTCOME_nightly_ui= \
  python3 tests/github-workflows/migration/generate_report.py | head -8

Observed on main at 56d522b:

# Langflow Migration Test Report
...
## Result: PASSED

For contrast, the same harness on run #115's shape (UI phase declared failure, recorded nothing) correctly yields FAILED + ## Unaccounted phases — that path is fixed and stays fixed.

Options

1. Reconcile against job.status. Pass it in alongside the phase outcomes; a red job whose report has no failure and no integrity entry is itself an integrity problem ("the runner failed this job outside every verified phase — read the log"). One env var, covers all present and future non-phase steps, including ones added later. Preferred.
2. id + PHASE_OUTCOME_* on the four steps. Precise attribution, but it only covers today's steps and silently degrades the moment someone adds a fifth.
3. A declared-phase manifest. The script knows the three phases must exist and flags any that is absent. Catches the gap from the other side, but reports "phase missing" where the real cause is an infra step — accurate but less useful than option 1's message.

Whichever lands, the fix keeps the existing property: nothing may conclude PASSED from missing data.

Done when

• A red job whose failure sits outside the three phases produces a report that does not say PASSED
• The message names where to look (job log / which step), instead of attributing the failure to a phase that never ran
• A unit test in tests/github-workflows/migration/test_generate_report.py reproduces the shape above and asserts the non-PASSED verdict — same style as the run-fix(playground): review playground-empty-message-send — known bug blocking @stable #115 regression test
• python-units stays green (uv run --with pytest --no-project python -m pytest tests/github-workflows/migration --ignore=tests/github-workflows/migration/test_ui_migration.py -q)
• Verified on a real workflow_dispatch run, or the residual risk stated explicitly in the PR