ci: fix E2E tests (#3855)

The scheduled E2E runs have been red since 2 July. Two independent
causes.

## The token

Every failure was `ApifyApiError: User was not found or authentication
token is not valid` (401), which looks like an expired secret but is not
— the login succeeds every run.

The harness read the token out of `~/.apify/auth.json`. Since
[apify-cli#1197](apify/apify-cli#1197) the
bundle distribution (what CI installs) stores the token in the OS
keyring and strips it from that file — intended behaviour, as plaintext
storage was the bug it fixed. The file still exists, so the `existsSync`
guard passed, `getApifyToken()` returned `undefined`, and
`process.env.APIFY_TOKEN ??= undefined` coerced it to the literal string
`"undefined"`. That bogus token is what the API rejected.

It landed within the `1.6.3` beta line (`beta.6` wrote `auth.json`,
`beta.25` wrote the keyring), which is why it broke ~2 weeks before
1.7.0 shipped.

- Pass `APIFY_TOKEN` from the existing secret, so the harness no longer
reaches into the CLI's credential store. The beta CLI is still installed
and exercised as before.
- Throw a clear error when `auth.json` has no token, instead of silently
yielding `"undefined"`.

## The demo store

With the token fixed, PLATFORM actually ran its tests for the first time
since 2 July — and `cheerio-robots-file` then failed on `429`s from
`warehouse-theme-metal.myshopify.com`. The robots.txt behaviour under
test was fine; the crawl never got off the ground, so `requestsFinished`
was 0 and the precondition failed.

The store rate-limits plain-HTTP clients from the platform's shared
egress IP. The browser-driven tests on the same IP get through. The six
tests crawling that store now go through
`Actor.createProxyConfiguration()`.

Author: B4nan

.github/workflows/test-e2e.yml

@@ -87,6 +87,8 @@ jobs:
                 run: yarn test:e2e
                 env:
                     STORAGE_IMPLEMENTATION: ${{ matrix.storage }}
+                    # Provided explicitly: the CLI may store the token in the OS keyring, where the tests can't read it.
+                    APIFY_TOKEN: ${{ secrets.APIFY_SCRAPER_TESTS_API_TOKEN }}
                     APIFY_HTTPBIN_TOKEN: ${{ secrets.APIFY_HTTPBIN_TOKEN }}
                     ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
                     GITHUB_TOKEN: ${{ github.token }}

test/e2e/adaptive-playwright-robots-file/actor/main.js

@@ -9,6 +9,8 @@ await Actor.init({
 });
 
 const crawler = new AdaptivePlaywrightCrawler({
+    // The store rate-limits the platform's shared egress IP, so crawl through a proxy.
+    proxyConfiguration: await Actor.createProxyConfiguration(),
     maxRequestsPerCrawl: 10,
     respectRobotsTxtFile: true,
     onSkippedRequest: (args) => crawler.log.warningOnce(`Request ${args.url} was skipped, reason: ${args.reason}`),

test/e2e/cheerio-robots-file/actor/main.js

@@ -9,6 +9,8 @@ await Actor.init({
 });
 
 const crawler = new CheerioCrawler({
+    // The store rate-limits the platform's shared egress IP, so crawl through a proxy.
+    proxyConfiguration: await Actor.createProxyConfiguration(),
     maxRequestsPerCrawl: 10,
     respectRobotsTxtFile: true,
 });

test/e2e/playwright-introduction-guide/actor/main.js

@@ -89,6 +89,8 @@ router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
 });
 
 const crawler = new PlaywrightCrawler({
+    // The store rate-limits the platform's shared egress IP, so crawl through a proxy.
+    proxyConfiguration: await Actor.createProxyConfiguration(),
     maxRequestsPerCrawl: 15, // so the test runs faster
     // Instead of the long requestHandler with
     // if clauses we provide a router instance.

test/e2e/playwright-robots-file/actor/main.js

@@ -9,6 +9,8 @@ await Actor.init({
 });
 
 const crawler = new PlaywrightCrawler({
+    // The store rate-limits the platform's shared egress IP, so crawl through a proxy.
+    proxyConfiguration: await Actor.createProxyConfiguration(),
     maxRequestsPerCrawl: 10,
     respectRobotsTxtFile: true,
 });

test/e2e/puppeteer-store-pagination-jquery/actor/main.js

@@ -11,6 +11,8 @@ const mainOptions = {
 
 await Actor.main(async () => {
     const crawler = new PuppeteerCrawler({
+        // The store rate-limits the platform's shared egress IP, so crawl through a proxy.
+        proxyConfiguration: await Actor.createProxyConfiguration(),
         maxRequestsPerCrawl: 10,
         preNavigationHooks: [
             async ({ page }, goToOptions) => {

test/e2e/puppeteer-store-pagination/actor/main.js

@@ -9,6 +9,8 @@ await Actor.init({
 });
 
 const crawler = new PuppeteerCrawler({
+    // The store rate-limits the platform's shared egress IP, so crawl through a proxy.
+    proxyConfiguration: await Actor.createProxyConfiguration(),
     maxRequestsPerCrawl: 10,
     preNavigationHooks: [
         async ({ page }, goToOptions) => {

test/e2e/tools.mjs

@@ -339,6 +339,15 @@ export async function getApifyToken() {
     }
 
     const { token } = await fs.readJSON(authPath);
+
+    // Newer CLI versions keep the token in the OS keyring, leaving auth.json without it.
+    if (!token) {
+        throw new Error(
+            'Your Apify token is not stored in auth.json (the CLI likely saved it in the OS keyring). ' +
+                'Set the "APIFY_TOKEN" environment variable, or re-run "apify login" with "APIFY_DISABLE_KEYRING=1".',
+        );
+    }
+
     return token;
 }