fix: stagger e2e batch fan-out into waves to avoid CodeBuild batch-orchestration fault - #3491
Closed
Simone319 wants to merge 2 commits into
Closed
fix: stagger e2e batch fan-out into waves to avoid CodeBuild batch-orchestration fault#3491Simone319 wants to merge 2 commits into
Simone319 wants to merge 2 commits into
Conversation
…chestration fault
The e2e_workflow CodeBuild batch deterministically FAULTed ("Internal
Service Error") ~22s after publish_to_local_registry because all ~184
generated e2e jobs depended solely on publish_to_local_registry and were
released to the batch orchestrator in one simultaneous wave.
Stagger the fan-out into sequential waves of at most WAVE_SIZE (46) jobs:
wave 0 depends on publish_to_local_registry; every later job depends on the
job WAVE_SIZE positions earlier. Each edge points to a strictly smaller
index, so the graph stays acyclic and no single completion event releases
more than ~46 jobs (down from ~184). cleanup_e2e_resources now depends on
the leaf jobs (last wave) so it still runs after every test.
This does not change which tests run or their isolation -- only the release
timing. Tradeoff: waves serialize, adding some wall-clock latency.
Contributor
Author
|
Closing as the fix is applied in #3497. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
amplify-category-api-e2e-workflowCodeBuild batch deterministically FAULTs withInternal Service Error("CodeBuild is experiencing issues") at theIN_PROGRESSbatch-orchestration phase, ~22s afterpublish_to_local_registrycompletes. All downstream test groups are force-STOPPED. This has blocked e2e since ~May 22, 2026.Root cause
The generated batch build-graph fans out into ~184 test jobs that all depend solely on
publish_to_local_registry, so they are released in a single simultaneous wave. CodeBuild's batch orchestrator faults on that burst.This is not a concurrency/quota issue (every sampled build had QUEUED=0s and provisioned immediately;
maximumBuildsAllowed=300 > 184) and not caused by test growth (the fan-out has been ~180-193 since Nov 2024, and was higher — 205-229 — in 2024 with no faults). Evidence points to a CodeBuild service-side change ~May 21-22, 2026 that reduced batch-orchestration tolerance: the identical 193-group batch ran healthy on May 19 and faulted on May 22 with no repo change. The pr-workflow (only ~11 groups) is unaffected.Fix
Stagger the fan-out into sequential waves in
scripts/split-e2e-tests.tsso the orchestrator never sees a single >~50-job release burst:WAVE_SIZE = 46.publish_to_local_registry; each later jobidepends on jobi - WAVE_SIZE(strictly-decreasing index → acyclic; max single-event release width = WAVE_SIZE).cleanup_e2e_resourcesnow depends on the final wave's leaf jobs so it still runs after all tests.codebuild_specs/e2e_workflow.yml.No test logic, test isolation, or job count changes — only
depend-onedges. Total jobs unchanged (192); max simultaneous release width 174 → 46.Verification
Triggered an e2e batch on this branch (
amplify-category-api-e2e-workflow:a6fe9799-...): afterpublish_to_local_registrysucceeded, wave 1 released ~46-49 jobs and the batch stayedIN_PROGRESSwith no FAULT — clearing the +22s death window that killed every prior unstaggered batch, and holding healthy through the run.Tradeoff
Waves serialize, adding some wall-clock latency — an accepted tradeoff to eliminate the deterministic orchestration FAULT.
Note
The underlying trigger is a CodeBuild service-side regression (~May 22, 2026); this change is a defensive workaround that makes the batch resilient to the lower orchestration ceiling. Worth a CodeBuild support ticket separately.