Handling Allowed and Disallowed Urls from Robots.txt

[PedroFonsecaDEV]

As an example lets look at the GitHub Robots.txt file.

I could not find any documentation on how to handle allowed and disallowed urls.

Especially cases like:

Disallow: /*tab=*
Allow: /*?tab=achievements&achievement=*

Could you please provide us an code example ?

Thanks

[B4nan]

See #229 (comment)

[PedroFonsecaDEV]

Hi @B4nan, yah, I had seen the comment before, but I couldn't find an example on how to use the robots parser and crawlee together. Could you please provide us with a basic example ?

For instance how to use the robot parser in combination with await enqueueLinks() ?

[B4nan]

You could use preNavigationHooks to skip pages based on it:

https://crawlee.dev/api/browser-crawler/interface/BrowserCrawlerOptions#preNavigationHooks

If you want to do it earlier, at the enqueueLinks level, you can use transformRequestFunction option (if you return a falsy value from it, the request will be skipped):

https://crawlee.dev/api/core/interface/EnqueueLinksOptions#transformRequestFunction