playwrightUtils for blocking patterns

[tsrseerist]

I am currently using https://crawlee.dev/api/playwright-crawler/namespace/playwrightUtils#blockRequests and the extraUrlPatterns to block certain urls. Some of these urls are ones like www.googletagmanager.com or images.taboola.com. However in my stats with my proxy provider it seems my scraper is still letting these through. Am I doing something wrong or does this not work the way I am thinking it does? I recall blocking stuff more effectively in puppeteer and have tried a few things in playwright but none seem to really work correctly. Any ideas?

[janbuchar]

Hi @tsrseerist and thanks for your interest in Crawlee! Could you please provide a minimal code snippet that reproduces the error so that we can troubleshoot this more efficiently?

[tsrseerist]

Sure:

import { createPlaywrightRouter, playwrightUtils, log } from "crawlee";

const patterns = ["http://www.googletagmanager.com/", "images.taboola.com"]

// in a route
await playwrightUtils.blockRequests(page, {
    extraUrlPatterns: patterns,
  });

I use a route for doing my scraping but this is basically all I am doing.

[janbuchar]

Could you also provide a crawler definition, please? And the page that you're trying to scrape?

[janbuchar]

Can you try calling blockRequests in a pre-navigation hook (https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#preNavigationHooks) instead of a request handler?

[tsrseerist]

yea I can try that, how does that look, something like?:

preNavigationHooks: [
    async (crawlingContext) => {
        const { page } = crawlingContext;
        await playwrightUtils.blockRequests(page, {
          extraUrlPatterns: patterns,
        });
    },
]

[tsrseerist]

Ok I tried this:

preNavigationHooks: [
        async ({ page }) => {
          await playwrightUtils.blockRequests(page, {
            extraUrlPatterns: patterns,
          });
        },
      ],

and it works exactly as expected so thank you for pointing me in the right direction.

[tsrseerist]

Sure:

import { PlaywrightCrawler, log } from 'crawlee';
import { router } from './routes.js';

const crawler = new PlaywrightCrawler({
    requestHandler: router,
});

await crawler.run(['https://cnn.com']);

There's really not much to it, just something basic like this should work. Also let me know if I am using extraUrlPatterns the correct way to.