Crawlers are opening incognito contexts regardless of configuration

[cybairfly]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/browser (BrowserCrawler)

Issue description

Injection of localStorage produces error:
DOMException: Failed to read the 'localStorage' property from 'Window': Access is denied for this document.

Investigation is pointing to https://www.chromium.org/for-testers/bug-reporting-guidelines/uncaught-securityerror-failed-to-read-the-localstorage-property-from-window-access-is-denied-for-this-document

PlaywrightCrawler is opening incognito contexts for both Chrome and Firefox. Using the option optionaluseIncognitoPages is not helping.

Code sample

const crawler = new PlaywrightCrawler({
    preNavigationHooks: [
        async (crawlingContext, gotoOptions) => {
            const {page} = crawlingContext;
            await page.context().addCookies(cookies.map(cookie => ({...cookie, sameSite: 'None'})));

            {
            const data = storage;
            const code = items =>
                Object
                    .entries(items)
                    .forEach(([key, value]) => localStorage[key] = value);

            await page.evaluate(code, data).catch(error => console.warn(error.message));
            }
        },
    ],
    postNavigationHooks: [
        async (crawlingContext, gotoOptions) => {
            await crawlingContext.closeCookieModals();
        }
    ],
    headless: false,
    // proxyConfiguration: new ProxyConfiguration({ proxyUrls: ['...'] }),
    requestHandler: router,
    // Comment this option to scrape the full website.
    maxRequestsPerCrawl: 20,
    requestHandlerTimeoutSecs: 5 * 60,
    useSessionPool: true,
    persistCookiesPerSession: true,
    launchContext: {
        launcher: firefox,
        useChrome: true,
        useIncognitoPages: false
    },

Package version

latest

Node.js version

latest

Operating system

No response

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[vladfrangu]

Is there any way you could send us a minimum reproduction sample we can use to debug this further? 🙏

[cybairfly]

Sure.

// For more information, see https://crawlee.dev/
import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
import { firefox } from 'playwright';
import { router } from './routes.js';
import cookies from './cookies.json' assert { type: "json" }
import storage from './storage.json' assert { type: "json" }



const crawler = new PlaywrightCrawler({
    preNavigationHooks: [
        async (crawlingContext, gotoOptions) => {
            const {page} = crawlingContext;
            await page.context().addCookies(cookies.map(cookie => ({...cookie, sameSite: 'None'})));

            {
            const data = storage;
            const code = items =>
                Object
                    .entries(items)
                    .forEach(([key, value]) => localStorage[key] = value);

            await page.evaluate(code, data).catch(error => console.warn(error.message));
            }

        },
    ],
    postNavigationHooks: [
        async (crawlingContext, gotoOptions) => {
            await crawlingContext.closeCookieModals();
        }
    ],
    headless: false,
    // proxyConfiguration: new ProxyConfiguration({ proxyUrls: ['...'] }),
    requestHandler: router,
    // Comment this option to scrape the full website.
    maxRequestsPerCrawl: 20,
    requestHandlerTimeoutSecs: 5 * 60,
    useSessionPool: true,
    persistCookiesPerSession: true,
    launchContext: {
        launcher: firefox,
        useChrome: true,
        useIncognitoPages: false
    },
});

await crawler.run(startUrls);

[vladfrangu]

Maybe something smaller? Or..if its easier for you, a GitHub repository we can clone? Either way we'll take a look

[cybairfly]

Great - thanks. It's not really dependent on any specific code. The issue simply is that crawlers are using incognito contexts with cookies and other site data features disabled apparently. The question is how to change that.

[B4nan]

FWIW, useIncognitoPages defaults to false, I doubt your problem is about that.

[cybairfly]

Indeed. Desperate attempt to induce some change in behavior.

[B4nan]

Don't you need to use page.evaluate here to actually execute the code in the browser? As that is where the localStorage object lives.

[cybairfly]

Yes, there is page.evaluate. The problem is it's evaluating within an incognito context with access to these window props disabled

[B4nan]

And another suspicious thing, why would you use useChrome: true with firefox?

launcher: firefox,
useChrome: true,

[B4nan]

Maybe firefox opens in incognito context by default, not sure about that. Does it work when you actually use chrome?

[cybairfly]

Have tried shuffling various different properties around so there might be some leftovers but pretty sure that has no effect here.

[cybairfly]

Doesn't matter which browser is used. The problem seems to be at a higher level - browserPool most likely.

[B4nan]

Well, I am more than sure that what you say is not true with chrome, we would be well aware if we open in incognito by default, as that hurts performance badly (~50% overhead).

[cybairfly]

Negative on chrome. Same thing across all browsers.