Skip to content

build: Deploy production closures without pnpm deploy --legacy (no-changelog) - #37890

Open
shortstacked wants to merge 3 commits into
masterfrom
devp-440-pnpm-deploy-speed
Open

build: Deploy production closures without pnpm deploy --legacy (no-changelog)#37890
shortstacked wants to merge 3 commits into
masterfrom
devp-440-pnpm-deploy-speed

Conversation

@shortstacked

@shortstacked shortstacked commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Deploys the n8n, task-runner and engine production closures with pnpm's dedicated-lockfile deploy instead of pnpm deploy --legacy. Supersedes #37866, tracked in DEVP-440.

The legacy deploy re-resolves the workspace against the registry on every build (~900MB of metadata on the cold cache every CI Docker job starts with). The dedicated lockfile is derived from pnpm-lock.yaml and mostly copies files.

CI, Docker Build Smoke Test (no cache), master vs this PR

amd64 arm64
task-runner deploy 17–25s → 3s 53–58s → 8s
n8n deploy + strips 43–67s → 41s 122–125s → 126s
Package deployment total 61–95s → 46s 179–187s → 138s

The n8n deploy is now bound by materializing ~113k files, not by resolution. Image size is unchanged: n8nio/n8n 1.86GB → 1.87GB, n8nio/runners 844MB → 844MB.

Changes

  • scripts/build-n8n.mjs — both deploys use --prod --config.inject-workspace-packages=true. The workspace-package glob matches pnpm's naming on Linux and macOS. Adds a timer for the task-runner deploy. Drops pnpm's deploy-only files (pnpm-lock.yaml, pnpm-workspace.yaml, workspace state) from compiled/ so cdxgen keeps inventorying node_modules for the SBOM.
  • packages/frontend/editor-uidependenciesdevDependencies. The package ships a Vite bundle and its published manifest already has no dependencies. A lockfile-derived closure ignores the build-time manifest trim, so this is what keeps the frontend libraries out of the image. The trim call is removed from the build; release-publish.yml still uses the script. Lint allows devDependency imports across the package; the module scaffolder anchors on devDependencies.
  • packages/@n8n/typeormpg-native declared as a devDependency. autoInstallPeers records the optional peer under dependencies, which pulled pg-native + libpq into the closure. TypeORM switches to the native driver when pg-native loads.
  • Task-runner closure stays extendable with pnpm add (the runners Dockerfiles add moment; users extend derived images per the docs). The raw deploy output is not: its manifest carries file: and peer-suffixed specifiers and pnpm prunes the whole runner on the next pnpm add (+4 -438). The build restores the source specifiers and records the closure under packages/@n8n/task-runner with an empty root importer — the shape the legacy deploy wrote.
  • docker/images/runners/Dockerfile — the runtime pnpm symlink now points at bin/pnpm.mjs. It pointed at bin/pnpm.cjs, which ships without the execute bit, so pnpm in the published image fails with Permission denied (pnpm: Permission denied #37808, since build: Install pnpm from npm in the runners images (no-changelog) #36702).
  • scripts/smoke-n8n-image.mjs — the workspace-package check matches the new naming and no longer allows the @n8n/ai-utilities double copy (the new deploy materializes every workspace package once). New check: pnpm add uuid in n8nio/runners:local must extend the closure and keep the runner resolving. It fails on the current master image and passes here.
  • packages/@n8n/engine — same deploy flags; deploy-only pnpm files removed from compiled/.

Not changed

@langchain/core still resolves to two peer variants in the n8n closure; master has the same two copies and the verifier stays report-only. pnpm 12 was benchmarked: no deploy speed gain over this, rejects the n8n closure with ERR_PNPM_DEPLOY_AMBIGUOUS_PEER (vue peer variants that differ only by a dev-only typescript alias), and leaves an unreferenced 173MB tree on macOS. That is a separate upgrade decision.

How to test

  1. pnpm build:docker — expect ✅ Task runner deployment completed in …s, ✅ Task runner closure is extendable, closure ~1.3GB / ~113k files.
  2. pnpm build:docker:smoke (needs helm) — all checks pass, including pnpm add extends the JS runner in n8nio/runners:local.
  3. docker run --rm --entrypoint sh n8nio/runners:local -c 'pnpm --version' prints 11.25.0.
  4. Start n8nio/n8n:local and open the editor.
  5. Optional: N8N_GENERATE_LICENSES=true node scripts/build-n8n.mjs renders the SBOM and THIRD_PARTY_LICENSES.md (1489 components, all licenses valid).

Related Linear tickets, Github issues, and Community forum posts

https://linear.app/n8n/issue/DEVP-440
fixes #37808

Review / Merge checklist

  • I have seen this code, I have run this code, and I take responsibility for this code.
  • PR title and summary are descriptive. (conventions)
  • Docs updated or follow-up ticket created.
  • Tests included.
  • PR Labeled with Backport to Beta, Backport to Stable, or Backport to v1 (if the PR is an urgent fix that needs to be backported)

🤖 PR Summary generated by AI

Use the dedicated-lockfile deploy with command-scoped workspace injection
for the n8n, task-runner and engine closures. The legacy deploy re-resolves
the workspace against the registry on every build; the dedicated lockfile
is derived from pnpm-lock.yaml.

Move editor-ui dependencies to devDependencies: the package ships a bundle
and its dependencies must stay out of the lockfile-derived closure. Drop the
build-time manifest trim, which the lockfile-derived deploy does not read.
Declare pg-native as a typeorm devDependency so the auto-installed optional
peer stays out of the production closure.

Keep the task-runner closure extendable with pnpm add: restore the source
specifiers, record the closure under its workspace importer, and remove the
pnpm files that point at this checkout. Link the runtime pnpm to the
executable bin/pnpm.mjs.
@cubic-dev-ai

cubic-dev-ai Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Running ultrareview automatically — Cross-cutting rewrite of production closure deployment for the n8n/engine/runners Docker images, moving runtime dependencies to devDependencies and changing pnpm deploy semantics—any missed dependency or peer-resolution change could break the built images at runtime.. I'll post findings when complete.

@n8n-assistant

n8n-assistant Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

PR review overview

Based on ownership of the 10 changed files in this PR:

Ownership Files owned Share Source code Test files Misc
@n8n-io/catalysts 5 50% +7 / -5 +3 / -3 +169 / -170
@n8n-io/qa-dx 3 30% +145 / -37 +0 / -0 +0 / -0
@n8n-io/frontend 2 20% +63 / -66 +0 / -0 +0 / -0
Total 10 100% +215 / -108 +3 / -3 +169 / -170

@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Bundle Report

Bundle size has no change ✅

Affected Assets, Files, and Routes:

view changes for bundle: editor-ui-esm

Assets Changed:

Asset Name Size Change Total Size Change (%)
assets/worker-*.js 3.2MB 3.22MB 17352.85% ⚠️
assets/worker-*.js -3.2MB 18.44kB -99.43%

@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

cubic-dev-ai[bot]

This comment was marked as outdated.

… test

The dedicated-lockfile deploy names a workspace package after its absolute
file: URL and materializes each one once, including @n8n/ai-utilities.
Allow devDependency imports across editor-ui: the package bundles everything
it imports, so all of its dependencies are devDependencies now.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Shadow auto-approve: would not auto-approve. Auto-approval blocked by 3 unresolved issues from previous reviews.

Re-trigger cubic

… deploy change

The frontend module scaffolder anchored on editor-ui's dependencies block,
which is now devDependencies. The engine image copies the whole deploy
output, so drop pnpm's deploy-only files there too. Add an image smoke
check that extends the JS runner with pnpm add, the flow derived images
use, so a deploy that makes pnpm prune the runner fails the build.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 4 files (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Shadow auto-approve: would not auto-approve. Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic

@n8n-assistant n8n-assistant Bot added the n8n team Authored by the n8n team label Sep 4, 2026
@shortstacked
shortstacked marked this pull request as ready for review September 4, 2026 17:14
@shortstacked
shortstacked requested review from a team and Matsuuu and removed request for a team September 4, 2026 17:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed n8n team Authored by the n8n team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pnpm: Permission denied

1 participant