Deploy large GraphQL APIs in CloudFormation-safe stack waves - #3484
Deploy large GraphQL APIs in CloudFormation-safe stack waves#3484MarlonJD wants to merge 17 commits into
Conversation
MarlonJD
changed the title
MarlonJD
changed the title
|
PTAL @svidgen |
|
Hi team any news on this? I think this fix would solve my issue as well! |
|
I also added a standalone real Amplify Gen2 repro project that reviewers can use to validate the published preview packages end-to-end: https://github.com/MarlonJD/amplify-gen2-stack-sharding-repro The sample project installs the patched packages through npm aliases/overrides:
It is a real Amplify Gen2 backend with one Validation flow: nvm use
npm install
npm run verify:packages
AWS_REGION=eu-central-1 MODEL_COUNT=80 npm run sandbox
npm run test:graphql
npm run sandbox:deleteWhat this validates:
I also verified locally that the repro project passes: npm run verify:packages
npx tsc --noEmit
npm ls @aws-amplify/data-construct @aws-amplify/graphql-api-construct @marlonjd/graphql-api-construct --all |
|
Hey @MarlonJD , looks promising. |
|
Thanks for raising this. I ran a bit more validation after your comment. What I verified so far:
The existing tests cover the fresh large-deploy path, grouped default DynamoDB tables, grouped That said, I agree the existing-app upgrade path needs one more explicit validation before we call it safe. The synth probe shows that some existing generated tables could move into a new stack group once sharding kicks in, so the next check should be a real deploy -> upgrade-to-preview -> CloudFormation change set/deploy test to verify that existing DynamoDB tables are not replaced or deleted. Do you have a preferred migration scenario you’d like me to prioritize for that validation? |
|
@MarlonJD thanks for the update! |
Thanks! The most useful help would be validating the existing-app upgrade path on a real Gen2 project, ideally in a non-production sandbox or copied environment. What I’m trying to confirm is whether applying the preview packages to an existing deployed app causes any unsafe CloudFormation changes, especially:
If you have a large enough existing project, even a sanitized/internal test copy would help. The ideal flow would be:
If you can share results, the most helpful details would be the approximate schema shape: number of models, indexes, relationships, auth rules, whether you use |
|
Hope this will be resolved ASAP. Have similar problem and waiting on solution |
|
@MarlonJD I tried your preview. I first tried to run it with only the number of resources in Then I tried to return removed queries/mutations, where the limit hits and got |
|
Also want to point out that the name of the autogenerated function directive stack by the amplify is Check out the end of the name, it's I suppose the reason for this is the stack name became too large so it's slightly shortened by the amplify framework. 
Hope it will be useful for you |
|
Thanks for the extra detail on the physical nested stack name. I checked this path too: the deployed I also reproduced the reported I added e2e coverage for this path in this PR and validated the published npm preview packages in a clean project. Could you retry the current PR preview with this override set? {
"overrides": {
"@aws-amplify/data-construct": "npm:@marlonjd/data-construct@1.17.4-auto-shard-transformer-stacks.5",
"@aws-amplify/graphql-api-construct": "npm:@marlonjd/graphql-api-construct@1.21.4-auto-shard-transformer-stacks.5",
"@aws-amplify/graphql-transformer": "npm:@marlonjd/graphql-transformer-stack-limits@2.4.2-stack-limits.2",
"@aws-amplify/graphql-transformer-core": "npm:@marlonjd/graphql-transformer-core-stack-limits@3.5.1-stack-limits.5",
"@aws-amplify/graphql-transformer-interfaces": "npm:@marlonjd/graphql-transformer-interfaces-stack-limits@4.3.0-stack-limits.2",
"@aws-amplify/graphql-model-transformer": "npm:@marlonjd/graphql-model-transformer-stack-limits@3.4.1-stack-limits.2",
"@aws-amplify/graphql-relational-transformer": "npm:@marlonjd/graphql-relational-transformer-stack-limits@3.1.13-stack-limits.4"
}
}After updating overrides, please reinstall/regenerate the lockfile and run |
|
Great. I will try it right now |
|
@MarlonJD Still got Did the clean install. Removed node_modules + lock file. Added overrides that you provided in the comment above. Runned npm i + sandbox |
|
I may try to lower the number of operations below the limit; run successful sandbox; add operations above the limit; try to run sandbox once again |
|
Rerun the logic as specified in the comment above and the error persists. |
|
Thanks for retrying and for the exact error. I rechecked this in a fresh Gen2-style project using the same override set from my previous comment, with In that clean install, a raw schema with 86 root
So the remaining failure you are seeing looks most likely like the sandbox process is still resolving an older/unpatched construct tree somewhere, rather than this exact preview package set being applied to the running Could you run these from the same project directory where npm ls @aws-amplify/backend @aws-amplify/backend-cli @aws-amplify/backend-data @aws-amplify/data-construct @aws-amplify/graphql-api-construct @aws-amplify/graphql-transformer-core --allnode -e 'const fs=require("fs"); const names=["@aws-amplify/data-construct","@aws-amplify/graphql-api-construct","@aws-amplify/graphql-transformer-core","@aws-amplify/data-construct/node_modules/@aws-amplify/graphql-transformer-core","@aws-amplify/graphql-api-construct/node_modules/@aws-amplify/graphql-transformer-core"]; for (const name of names) { try { const path=require.resolve(name + "/package.json"); const p=require(path); console.log(name, "=>", p.name, p.version, path); } catch (e) { console.log(name, "MISSING", e.code); } } for (const name of ["@aws-amplify/graphql-transformer-core/lib/transformer-context/stack-manager.js","@aws-amplify/data-construct/node_modules/@aws-amplify/graphql-transformer-core/lib/transformer-context/stack-manager.js","@aws-amplify/graphql-api-construct/node_modules/@aws-amplify/graphql-transformer-core/lib/transformer-context/stack-manager.js"]) { try { const path=require.resolve(name); const src=fs.readFileSync(path,"utf8"); console.log("STACK_MANAGER", name, { path, patched: src.includes("getStackNameWithinResourceBudget"), autoLimit: src.includes("DEFAULT_AUTO_STACK_RESOURCE_ESTIMATE = 400") }); } catch (e) { console.log("STACK_MANAGER", name, "MISSING", e.code); } }'One note: the bundled nested Also, please use the project-local CLI path when retrying, for example: npx --no-install ampx sandboxThat avoids accidentally running a freshly downloaded/global |
|
@MarlonJD node -e command returned: Regarding cli path, I don't have ampx installed globally, it's only project specific |
|
Thanks, this output confirms the preview packages are not being applied yet. The expected output should show:
Your output still shows the current published packages:
So the sandbox is still running against the unpatched dependency tree. Could you check that the Please run: npm --version
npm pkg get overrides
npm explain @aws-amplify/data-construct
npm explain @aws-amplify/graphql-api-constructThen reinstall from that same root and re-run the package check. Once the output shows the |
|
Got I use monorepo via turborepo. My main backend is located as packages/-main-backend and I run sandbox command via -w "-main-backend" After all the changes, I run sandbox and got |
|
The error persists even if I decrease the number of resources in function directive to below 500 limit |
|
Thanks for the detailed repro. I found the root cause here: the previous stack sharding change could move AppSync API-scoped resources ( I have a fix ready so those AppSync resources stay in the original function directive stack for existing deployments, while only the generated Lambda datasource IAM roles/policies can overflow into sharded stacks. The large Update: I published a new preview package set with this fix: {
"overrides": {
"@aws-amplify/data-construct": "npm:@marlonjd/data-construct@1.17.4-auto-shard-transformer-stacks.6",
"@aws-amplify/graphql-api-construct": "npm:@marlonjd/graphql-api-construct@1.21.4-auto-shard-transformer-stacks.6",
"@aws-amplify/graphql-transformer-core": "npm:@marlonjd/graphql-transformer-core-stack-limits@3.5.1-stack-limits.6"
}
}Please add this I also verified the new override set in a clean temp npm project: |
|
Hi @MarlonJD . After your update, it seems to be working. I first run on stack with below limit and it already created a FunctionDirectiveStack2 where it put some of the operations. I had previously 475 resources in FunctionDirectiveStack. And after sandbox run I was decreased to 417 and other were migrated to FunctionDirectiveStack2. After I added additional queries/mutation, the count of resources on FunctionDirectiveStack remained the same (417) and the FunctionDirectiveStack2 got additional queries/mutations. |
|
What appears strange to me is that after adding even more queries/mutations I got 4 FunctionDirective stacks. Two of them have 51 resources each, one has 25 resources and the main big one has 399 resources. Can you give a little insight into how the division works? How I can predict in what stacks the resources were allocated? |
|
Thanks for testing and sharing the resource counts. I agree the distribution looks a bit surprising at first glance. The current split is not trying to evenly balance the final CloudFormation resource counts across the FunctionDirective stacks. It is more conservative than that. For update safety, the AppSync API-scoped resources, such as DataSource, FunctionConfiguration, and Resolver, are kept in the original FunctionDirectiveStack. Moving those between nested stacks can be risky for existing deployments. The overflow stacks mainly receive the generated IAM roles and policies that are safer to move. So a shape like 399 / 51 / 51 / 25 can happen: the large stack still contains the AppSync resources, while the smaller stacks contain the movable IAM resources. The exact split depends on generation order and the adaptive resource budget used during synth, so it is deterministic for a given schema/package version but not something users should rely on as a stable contract. That said, I agree this is not ideal from a predictability/readability perspective. I am looking into a better follow-up approach so the placement is easier to reason about and handles larger @function-heavy schemas more cleanly. |
|
Thank you for response. Also, for the work that was done. Hope this solution will be checked by the amplify team and added to the main package |
|
Can you also provide the information on how ready are those packages for the usage in production? Is there something that you plan to add/edit before? Right now we're pretty blocked on the development of the application, so we're discussing the idea of using your packages. Even though they're not merged yet. Can you give info on it? |
|
@Simone319 Please, take a look at this PR. It worked for me and hope we can prepare it to be published to the main packages |
b458f47 to
ffcd480
Compare
|
Are there any updates on the issue? Maybe there is an approximate timeline on when this feature is planned to be released? |
|
Update after the last
Additional validation completed:
Preview update:
Not run:
|
|
Hi Amplify team, Could someone please take a look at this PR when possible? The issue it addresses has been open for quite some time and continues to affect users in production environments. A review, feedback, or an update on next steps would be greatly appreciated. Thank you for your time and for maintaining Amplify. |
4 participants
Summary
This PR makes large generated GraphQL APIs deployable by splitting generated resources across CloudFormation-safe stack groups. It addresses the 1 MB template size limit, the 500 resources per stack limit, and the 2500 resources per CloudFormation operation limit.
Instead of keeping all generated nested stacks under one growing parent stack hierarchy, overflow generated stacks are promoted into ordered top-level stack groups. This keeps individual templates and stacks smaller, while also making CloudFormation process large APIs as multiple ordered stack operations.
Why
Large Gen2 GraphQL APIs can exceed multiple CloudFormation limits as the generated schema grows:
Reducing generated operations or removing secondary indexes can delay the issue, but it does not fix the underlying scaling problem. Large schemas still eventually generate too many AppSync resolvers, functions, data sources, IAM roles, DynamoDB tables, outputs, and nested stacks for a single generated stack layout.
The 500-resource stack limit and 1 MB template limit require generated resources to be split into smaller stacks. The 2500-resource operation limit also means that simply creating more nested stacks is not enough if they all remain under the same parent stack hierarchy. CloudFormation can still process the whole nested hierarchy as one oversized operation.
Addresses:
Problems found during validation
While testing this with a large Gen2 schema, a few additional issues appeared:
getGeneratedResourcescould no longer discover all generated constructs unless discovery was extended.Export.Namevalues invalid because CloudFormation export names cannot depend onFn::ImportValue,Fn::GetAtt, orFn::GetAZs.@functioncan createFunctionDirectiveStackbefore the stack manager later reuses that same stack for automatic sharding, which meant the stack could miss the default-stack metadata used by adaptive sizing.How this works
This change splits large generated APIs in two layers.
First, the transformer stack manager now auto-shards generated resources by an estimated resource budget. Explicit stack mappings are still respected, but unmapped/generated resources are distributed across additional generated stacks before a single stack grows too large.
The budget intentionally leaves headroom below CloudFormation's 500-resource stack limit because some logical transformer resources synthesize into multiple CloudFormation resources. For example, pipeline resolvers and Lambda data sources count as more than one estimated resource.
This helps keep each generated stack below both:
Second, the nested stack provider now keeps the first batch of generated nested stacks under the data stack, then promotes overflow batches into separate top-level CDK stacks named like:
AmplifyGraphqlApiStackGroup1AmplifyGraphqlApiStackGroup2AmplifyGraphqlApiStackGroup3Each group contains a bounded number of generated nested stacks. The groups are chained with dependencies:
This makes CloudFormation deploy them as separate ordered stack operations instead of one oversized nested-stack operation. That is the part that addresses the 2500-resource operation limit.
Additional fixes
getGeneratedResourcesto walk both the original generated scope and the generated top-level stack groups.exportNameis omitted.FunctionDirectiveStack.Validation
git diff --checkPATH=/Users/marlonjd/.nvm/versions/node/v20.18.3/bin:$PATH yarn workspace @aws-amplify/graphql-api-construct test nested-stack-provider.test.ts construct-exports.test.ts --runInBand --coverage=falsePATH=/Users/marlonjd/.nvm/versions/node/v20.18.3/bin:$PATH yarn workspace @aws-amplify/graphql-model-transformer test dynamo-model-resource-generator.test.ts --runInBand --coverage=falsePATH=/Users/marlonjd/.nvm/versions/node/v20.18.3/bin:$PATH yarn workspace @aws-amplify/graphql-transformer-core test stack-manager.test.ts --runInBand --coverage=falseI also validated this against a real Gen2 sandbox deployment with a 200 parent / 200 child model schema in
eu-central-1.The deployment created 8 top-level GraphQL API stack groups and completed successfully without hitting:
Export.Namevalidation errorThe smoke-test sandbox was deleted afterwards, and no matching CloudFormation stacks remained.
Trying this before merge
A preview build of this PR is available on npm for testing before it is merged.
For Gen2 projects, use the full override set below so the local project and bundled transformer packages resolve to the same preview build:
{ "overrides": { "@aws-amplify/data-construct": "npm:@marlonjd/data-construct@1.17.4-auto-shard-transformer-stacks.6", "@aws-amplify/graphql-api-construct": "npm:@marlonjd/graphql-api-construct@1.21.4-auto-shard-transformer-stacks.7", "@aws-amplify/graphql-transformer": "npm:@marlonjd/graphql-transformer-stack-limits@2.4.2-stack-limits.2", "@aws-amplify/graphql-transformer-core": "npm:@marlonjd/graphql-transformer-core-stack-limits@3.5.1-stack-limits.6", "@aws-amplify/graphql-transformer-interfaces": "npm:@marlonjd/graphql-transformer-interfaces-stack-limits@4.3.0-stack-limits.2", "@aws-amplify/graphql-model-transformer": "npm:@marlonjd/graphql-model-transformer-stack-limits@3.4.1-stack-limits.2", "@aws-amplify/graphql-relational-transformer": "npm:@marlonjd/graphql-relational-transformer-stack-limits@3.1.13-stack-limits.4" } }Then reinstall and deploy from the local project:
For lower-level testing, the individual patched packages are also published:
{ "overrides": { "@aws-amplify/data-construct": "npm:@marlonjd/data-construct@1.17.4-auto-shard-transformer-stacks.6", "@aws-amplify/graphql-api-construct": "npm:@marlonjd/graphql-api-construct@1.21.4-auto-shard-transformer-stacks.7", "@aws-amplify/graphql-transformer": "npm:@marlonjd/graphql-transformer-stack-limits@2.4.2-stack-limits.2", "@aws-amplify/graphql-transformer-core": "npm:@marlonjd/graphql-transformer-core-stack-limits@3.5.1-stack-limits.6", "@aws-amplify/graphql-transformer-interfaces": "npm:@marlonjd/graphql-transformer-interfaces-stack-limits@4.3.0-stack-limits.2", "@aws-amplify/graphql-model-transformer": "npm:@marlonjd/graphql-model-transformer-stack-limits@3.4.1-stack-limits.2", "@aws-amplify/graphql-relational-transformer": "npm:@marlonjd/graphql-relational-transformer-stack-limits@3.1.13-stack-limits.4" } }Make sure
ampxis resolved from the local project dependencies. Running a globally installedampxmay bypass these overrides.Additional Validation
Latest
@functionfield-auth edge-case update was validated with:git diff --checkenv npm_config_devdir=/private/tmp/node-gyp YARN_CACHE_FOLDER=/Users/marlonjd/.cache/yarn corepack yarn workspace @aws-amplify/graphql-transformer-core test stack-manager.test.ts --runInBand --coverage=falseenv npm_config_devdir=/private/tmp/node-gyp YARN_CACHE_FOLDER=/Users/marlonjd/.cache/yarn corepack yarn workspace @aws-amplify/graphql-function-transformer test amplify-graphql-function-transformer.test.ts --runInBand --coverage=false --silentenv npm_config_devdir=/private/tmp/node-gyp YARN_CACHE_FOLDER=/Users/marlonjd/.cache/yarn corepack yarn workspace @aws-amplify/graphql-api-construct test adaptive-sizer.test.ts --runInBand --coverage=falseenv npm_config_devdir=/private/tmp/node-gyp YARN_CACHE_FOLDER=/Users/marlonjd/.cache/yarn corepack yarn workspace amplify-graphql-api-construct-tests jest function-directive-stack-limits.test.ts --runInBand --coverage=falseenv npm_config_devdir=/private/tmp/node-gyp YARN_CACHE_FOLDER=/Users/marlonjd/.cache/yarn corepack yarn workspace amplify-graphql-api-construct-tests build-tests@aws-amplify/graphql-transformer-corestack-manager.jswithaddDefaultStackNameMetadata.@marlonjd/data-construct@1.17.4-auto-shard-transformer-stacks.5,@marlonjd/graphql-api-construct@1.21.4-auto-shard-transformer-stacks.5, and@marlonjd/graphql-transformer-core-stack-limits@3.5.1-stack-limits.5synthesized the reported@function+ IAM field-auth shape into twoFunctionDirectiveStacktemplates:300resources and216resources.stack-limits.4Data construct preview reproduced the failure: oneFunctionDirectiveStackwith516resources, exceeding the 500-resource CloudFormation stack limit.@marlonjd/graphql-transformer-core-stack-limits@3.5.1-stack-limits.6(stack-limits)@marlonjd/graphql-api-construct@1.21.4-auto-shard-transformer-stacks.7(auto-shard-transformer-stacks)@marlonjd/data-construct@1.17.4-auto-shard-transformer-stacks.6(auto-shard-transformer-stacks)The earlier edge-case hardening update was validated with:
git diff --checkyarn test stack-manager.test.ts --runInBand --coverage=falseinpackages/amplify-graphql-transformer-coreyarn test nested-stack-provider.test.ts adaptive-sizer.test.ts --runInBand --coverage=falseinpackages/amplify-graphql-api-constructyarn test resources.test.ts --runInBand --coverage=falseinpackages/amplify-graphql-api-constructyarn jest commands.test.ts --runInBand --coverage=false --reporters=default --env=node --testRunner=jest-circus/runnerinpackages/amplify-graphql-api-construct-testsyarn jest --runTestsByPath scripts/__tests__/verify-construct-dependencies.test.ts --runInBand --coverage=false --reporters=default --env=node --testRunner=jest-circus/runner --config jest.config.base.js --roots scripts --globals '{"ts-jest":{"diagnostics":false,"isolatedModules":true}}'yarn buildin the touched transformer/API construct packages, includingamplify-graphql-transformer-interfaces,amplify-graphql-transformer-core,amplify-graphql-transformer,amplify-graphql-api-construct, and the legacygraphql-transformer-coredependency needed by test buildsyarn build-testsinpackages/amplify-graphql-api-construct-testsyarn extract-apiinamplify-graphql-transformer-interfaces,amplify-graphql-transformer-core,amplify-graphql-transformer, andamplify-graphql-api-constructyarn verify-construct-dependenciesnpm installresolved the changed@aws-amplify/*packages to the@marlonjd/*-stack-limitsaliases, including the.4data construct, GraphQL API construct, transformer core, and relational transformer preview packages.@aws-amplify/data-constructwith the override set and synthesizing a publicTodo @modelschema completed successfully (15synthesized root resources).eu-central-1using stack prefixAmplifyPreviewE2EStackLimits-20260520-03: deployed both default DynamoDB andAMPLIFY_TABLEstrategies with owner/groups auth, relations, secondary indexes, grouped grants, adaptive grouped nested stacks, and Lambda grant targets.E2E_OPERATION_BUDGET=450, produced29nested stacks across33templates, verified max template resource count was48, and verified generated root/group operation resource counts stayed below the 2500 hard limit.npx cdk destroy --all --force; follow-up CloudFormationlist-stacksreturned no active or failed stacks for the prefix, and both AppSync APIs returnedNotFound.AMPLIFY_TABLEand grouped nested stacks; after narrowing the fixture serialization to root-parent operations and keepingAmplifyTableManagerfirst, the final deploy/destroy passed.graphql-api-constructbundle include the patchedStackManagerOptionsandestimatedResourceCountAPIs.The
resources.test.tsrun also covers grouped grant behavior for both default DynamoDB tables andAMPLIFY_TABLE, asserts no grouped table grant policy depends onFn::ImportValue/nested outputs, verifies index ARN grants remain present, checks grouped stream grants do not fabricate a stream ARN, and asserts generated direct/group operationCfnResourcecounts stay below the 2500 hard limit.Update-safety readiness refresh
After the previous
399/51/51/25validation snapshot, I tightened this PR around update safety and review readiness rather than broadening the feature surface.Safety decisions made
AWS::DynamoDB::TableandCustom::AmplifyDynamoDBTableresources are kept out of generated stack groups.generated-stack-helpers.tsand reused by adaptive sizing, construct exports, nested stack provider, and DynamoDB table wrapper paths.RED/GREEN evidence
ExistingModelCmoving unexpectedly.GREEN: root nested stack preservation now keeps existing stateful stacks under the original parent.
ConnectionStackand observedConnectionStack2.GREEN: stack manager now uses
pinned,movable, andunclassified-api-scopedplacement classification and fails closed for ambiguous API-scoped resources.GREEN: adaptive sizing now enforces both the 500-resource nested stack cap and the 2500-operation cap.
GREEN: the same package smoke passed with npm cache redirected to
/private/tmp/amplify-npm-cache.Verification run after the refresh
git diff --check@aws-amplify/graphql-api-constructfocused tests:nested-stack-provider.test.tsadaptive-sizer.test.tsupdate-safety-placement.test.tsresources.test.tsconstruct-exports.test.ts@aws-amplify/graphql-transformer-core:stack-manager.test.ts@aws-amplify/graphql-function-transformer:amplify-graphql-function-transformer.test.ts@aws-amplify/graphql-auth-transformer:get-scope-for-field.test.tsfunction-directive-stack-limits.test.tscommands.test.tsupdate-safety-change-set.test.tsamplify-graphql-api-construct-tests build-tests@aws-amplify/graphql-transformer-interfaces build@aws-amplify/graphql-transformer-core build@aws-amplify/graphql-transformer build@aws-amplify/graphql-api-construct buildcorepack yarn extract-apiacross 25 packages@aws-amplify/graphql-api-construct packagenpx cdk synth --app "node app.js"AmplifyPreviewUpdateSafety-20260619-localeu-central-1Preview package
@marlonjd/graphql-api-construct@1.21.4-auto-shard-transformer-stacks.7auto-shard-transformer-stackslatestwas not changed and remains1.21.4-auto-shard-transformer-stacks.4.Not run
corepack yarn cloud-e2erequiresscripts/.enve2e account configuration, and that file was not present locally.