Skip to content

Commit 20ba48c

Browse files
authored
docs: reflect the recent SessionPool changes in a guide (#3724)
Adds recent changes to the `SessionPool` and related classes to the existing guides. Closes #796
1 parent 19fa874 commit 20ba48c

3 files changed

Lines changed: 254 additions & 19 deletions

File tree

docs/guides/avoid_blocking.mdx

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ title: Avoid getting blocked
44
description: How to avoid getting blocked when scraping
55
---
66

7+
import ApiLink from '@site/src/components/ApiLink';
8+
79
import Tabs from '@theme/Tabs';
810
import TabItem from '@theme/TabItem';
911
import CodeBlock from '@theme/CodeBlock';
@@ -18,6 +20,8 @@ A scraper might get blocked for numerous reasons. Let's narrow it down to the tw
1820

1921
Browser fingerprint is a collection of browser attributes and significant features that can show if our browser is a bot or a real user. Moreover, most browsers have these unique features that allow the website to track the browser even within different IP addresses. This is the main reason why scrapers should change browser fingerprints while doing browser-based scraping. In return, it should significantly reduce the blocking.
2022

23+
The two are not handled separately. In Crawlee a <ApiLink to="core/class/Session">`Session`</ApiLink> ties an IP, a cookie jar, and a fingerprint together into one consistent identity, and the <ApiLink to="core/class/SessionPool">`SessionPool`</ApiLink> rotates those identities as a unit — so a fresh fingerprint always arrives with a fresh IP. This guide covers the fingerprint half; see the [session management guide](./session-management) for how to control the rotation, and the [proxy management guide](./proxy-management) for the IP half.
24+
2125
## Using browser fingerprints
2226

2327
Changing browser fingerprints can be a tedious job. Luckily, Crawlee provides this feature with zero configuration necessary - the usage of fingerprints is enabled by default and available in `PlaywrightCrawler` and `PuppeteerCrawler`. So whenever we build a scraper that is using one of these crawlers - the fingerprints are going to be generated for the default browser and the operating system out of the box.
@@ -56,6 +60,35 @@ On the contrary, sometimes we want to entirely disable the usage of browser fing
5660
</TabItem>
5761
</Tabs>
5862

63+
## Fingerprints for HTTP crawlers
64+
65+
Every session carries a lightweight fingerprint hint — a `browser`, `platform`, and `device` triple — that the request's HTTP client receives and applies on a best-effort basis.
66+
By default each session is given a realistic, randomized fingerprint (the host operating system as `platform`, with a plausible `browser`/`device` for it), and it rotates with the session just like the IP and cookies do.
67+
68+
How much of the hint is used depends on the client. The [`impit`](impit-http-client) HTTP client maps the session's `browser` hint to a matching TLS and HTTP impersonation profile,
69+
so the connection's low-level signature lines up with the headers being sent.
70+
71+
The *same* hint also drives browser crawlers, where it seeds the generated browser fingerprint. The hint only fixes the broad strokes — the browser family, operating system, and device — so a session presents a coherent profile, but it does not make the two backends produce byte-identical fingerprints: `impit` and a real browser will still differ in the finer details (a slightly different user-agent string, for example).
72+
73+
You can pin the fingerprint explicitly through `sessionOptions` when you need a specific profile:
74+
75+
```js
76+
import { CheerioCrawler, SessionPool } from 'crawlee';
77+
import { ImpitHttpClient } from '@crawlee/impit-client';
78+
79+
const crawler = new CheerioCrawler({
80+
httpClient: new ImpitHttpClient(),
81+
sessionPool: new SessionPool({
82+
sessionOptions: {
83+
fingerprint: { browser: 'firefox', platform: 'windows', device: 'desktop' },
84+
},
85+
}),
86+
requestHandler: async ({ $ }) => {
87+
// requests impersonate desktop Firefox on Windows
88+
},
89+
});
90+
```
91+
5992
## Camoufox
6093

6194
For some protections, using our integrated solutions is not enough, one example could be the Cloudflare challenge. For such pages, you can try [Camoufox](https://camoufox.com/), a custom stealthy build of Firefox for web scraping. It might not get you through the challenge automatically, but with our `handleCloudflareChallengeHook` post-navigation hook, it should be able to successfully mimic the required user action and get you through it. The hook also reloads the page after the challenge clears and propagates the fresh response back into the crawling context.

docs/guides/impit-http-client/impit-http-client.mdx

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,6 @@ import CheerioCrawlerSource from '!!raw-loader!./cheerio-crawler.ts';
1111
import HttpCrawlerSource from '!!raw-loader!./http-crawler.ts';
1212
import AdvancedConfigSource from '!!raw-loader!./advanced-config.ts';
1313

14-
## Introduction
15-
1614
The `ImpitHttpClient` is an HTTP client implementation based on the [Impit](https://github.com/apify/impit) library. It enables browser impersonation for HTTP requests, helping you bypass bot detection systems without running an actual browser.
1715

1816
:::info Successor to got-scraping

0 commit comments

Comments
 (0)