You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The daily instance reaper deletes strictly by name pattern and age. Its threshold is shorter than the timeout we grant long tests, and unlike the disk reaper it has no in-use check — so a long test that runs past the threshold gets its VM and state disk deleted out from under it, mid-test.
gcp-delete-old-instances.sh selects on name~-[0-9a-f]{7,}$ AND creationTimestamp < <3 days ago> and runs instances delete --delete-disks=all on every match. There is no predicate for "this instance is still doing work". Contrast gcp-delete-old-disks.sh, which includes -users:* and therefore only ever touches unattached disks — the instance reaper has no equivalent.
Why nothing has broken yet
This is latent, not active. The last successful sync-full-mainnet took ~20 hours (see #11232), far under the 72-hour threshold. But the 5-day timeout exists precisely because these tests are expected to sometimes run much longer, and the workflow's own naming block assumes instances can legitimately live for multi-day syncs. The two limits contradict each other, and the contradiction is invisible until a sync is slow enough to hit it.
What it would look like when it fires
At 07:00 UTC on day 3, the VM disappears mid-test. From the job's side: the SSH connection drops, docker wait hangs until timeout-minutes cancels the job, and the failure surfaces as a cancellation with no pointer at the reaper. That is the same hard-to-attribute shape as the runner-recycling loss in #11232 — anyone debugging it would look at runners and capacity, not at the cleanup cron. The state disk is deleted along with the instance (--delete-disks=all), so a multi-day sync's progress is unrecoverable.
Possible directions
Label-based exemption (cheapest): the deploy workflow already labels instances (app, test, environment, commit, since fix(ci): give GCP test resource names run identity #11364). Add long-test=true when is_long_test is set, and give the reaper a separate, longer threshold (e.g. 6 days, just past the 5-day timeout) for instances carrying it. Ordinary instances keep the 3-day limit, so cost stays bounded.
An in-use predicate, paralleling the disk reaper's -users:*: skip instances whose uptime is shorter than the threshold or that are actively running the test container. Harder to express in a gcloud filter; likely needs a describe per candidate.
Raise DELETE_INSTANCE_DAYS to 6 globally: simplest change, but every leaked non-long instance then costs 3 extra days of VM + 400 GB disk, which is what the short threshold exists to avoid.
The first option seems like the right trade-off, and became easy once #11364 made labels the carrier for instance metadata.
The daily instance reaper deletes strictly by name pattern and age. Its threshold is shorter than the timeout we grant long tests, and unlike the disk reaper it has no in-use check — so a long test that runs past the threshold gets its VM and state disk deleted out from under it, mid-test.
The numbers
DELETE_INSTANCE_DAYS(zfnd-delete-gcp-resources.yml)timeout-minuteswithis_long_test: truesync-to-mandatory-checkpoint,sync-full-mainnet,sync-full-testnet,lwd-sync-fullgcp-delete-old-instances.shselects onname~-[0-9a-f]{7,}$ AND creationTimestamp < <3 days ago>and runsinstances delete --delete-disks=allon every match. There is no predicate for "this instance is still doing work". Contrastgcp-delete-old-disks.sh, which includes-users:*and therefore only ever touches unattached disks — the instance reaper has no equivalent.Why nothing has broken yet
This is latent, not active. The last successful
sync-full-mainnettook ~20 hours (see #11232), far under the 72-hour threshold. But the 5-day timeout exists precisely because these tests are expected to sometimes run much longer, and the workflow's own naming block assumes instances can legitimately live for multi-day syncs. The two limits contradict each other, and the contradiction is invisible until a sync is slow enough to hit it.What it would look like when it fires
At 07:00 UTC on day 3, the VM disappears mid-test. From the job's side: the SSH connection drops,
docker waithangs untiltimeout-minutescancels the job, and the failure surfaces as a cancellation with no pointer at the reaper. That is the same hard-to-attribute shape as the runner-recycling loss in #11232 — anyone debugging it would look at runners and capacity, not at the cleanup cron. The state disk is deleted along with the instance (--delete-disks=all), so a multi-day sync's progress is unrecoverable.Possible directions
app,test,environment,commit, since fix(ci): give GCP test resource names run identity #11364). Addlong-test=truewhenis_long_testis set, and give the reaper a separate, longer threshold (e.g. 6 days, just past the 5-day timeout) for instances carrying it. Ordinary instances keep the 3-day limit, so cost stays bounded.-users:*: skip instances whose uptime is shorter than the threshold or that are actively running the test container. Harder to express in agcloudfilter; likely needs a describe per candidate.DELETE_INSTANCE_DAYSto 6 globally: simplest change, but every leaked non-long instance then costs 3 extra days of VM + 400 GB disk, which is what the short threshold exists to avoid.The first option seems like the right trade-off, and became easy once #11364 made labels the carrier for instance metadata.
Related