test: repair E2E test infrastructure for pnpm + crawlee v4#3805
Conversation
janbuchar
added
adhoc
The published apify SDK depends on the 3.x line of @crawlee/*, but the
e2e tests run it against the local 4.x workspace packages. Its runtime
Crawlee-version guard tripped for the in-process LOCAL/MEMORY runs
("Detected incompatible Crawlee version"). Override apify's @crawlee/core,
@crawlee/types and @crawlee/utils to the workspace versions, mirroring the
per-actor overrides.apify used for PLATFORM tests.
apify@3.x is not runtime compatible with @crawlee/core@4 (it iterates Configuration.INTEGER_VARS, which was removed in v4), causing "TypeError: Configuration.INTEGER_VARS is not iterable". Pin the root apify to the v4 beta and point the e2e actors at the next-v4 dist-tag so both the in-process (LOCAL/MEMORY) harness and the pushed PLATFORM actors run the matching v4 SDK.
janbuchar
changed the title
B4nan
changed the title
|
btw the new |
barjin
left a comment
barjin
left a comment
There was a problem hiding this comment.
lgtm, thank you @janbuchar !
I also got a brainwave while reading this ⬇️
| // npm (used when building the actor on the platform) does not understand pnpm's | ||
| // `workspace:` protocol, so rewrite internal `@crawlee/*` deps to sibling `file:` | ||
| // references and make sure those packages get copied as well. | ||
| for (const depGroup of ['dependencies', 'optionalDependencies', 'peerDependencies']) { |
There was a problem hiding this comment.
I'm wondering whether we could use pnpm deploy (docs) when building the e2e actors somehow.
We could essentially save ourselves all this setup with something like
pnpm deploy --filter=test-cheerio-default distwhich generates dist folder with the test-cheerio-default workspace package contents + injects node_modules with the right versions of workspace packages etc. This can then be directly copied into the Docker image.
Granted, this relies on some preconditions (e.g., the e2e tests are workspace packages), but these can be imo done dynamically with less code.
Anyway, not a blocker rn, but maybe a way we can let Claude explore later on?
There was a problem hiding this comment.
That sounds amazing. The e2e setup is far away from great, and now this PR makes things worse (more code for the same functionality). I would love to get rid of this complexity.
4 participants
Fixes the E2E test harness, which was crashing at startup and never actually running tests.
Changes
crawlee+ all e2e-imported@crawlee/*packages as root devDeps so in-process (LOCAL/MEMORY) actor imports resolve under pnpm.copyPackages: rewriteworkspace:*→file:and copy transitive deps, so pushed PLATFORM actors build with npm.yarn/npxforpnpm/pnpm exec(storage-local install, camoufox fetch) to satisfy thepackageManager/devEnginesgate.apifyto the v4 SDK (root override + actors →next-v4); the 3.x SDK is runtime-incompatible with@crawlee/core@4.Result
Harness now runs; MEMORY passes 19 tests. Remaining failures are not infra:
@apify/storage-localdoesn't implement the v4StorageClientinterface (createRequestQueueClientetc.) — needs a v4-compatible storage-local or dropping the job.crawler.log.setLevel,contextPipelineBuilder, assorted assertions).