Skip to content

Commit 73ad18b

Browse files
D4VinciD4Vinci
committed
refactor(mcp)!: More reliable structure + new tools
A better fix than #418 is to eliminate this bug class and keep the AI agent knowing the default parameter values. I have also shortened the docstrings where possible to save tokens.
1 parent d4e121c commit 73ad18b

6 files changed

Lines changed: 472 additions & 459 deletions

File tree

‎agent-skill/Scrapling-Skill.zip‎

367 Bytes
Binary file not shown.

‎agent-skill/Scrapling-Skill/references/mcp-server.md‎

Lines changed: 39 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# Scrapling MCP Server
22

3-
The Scrapling MCP server exposes ten tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, and page screenshots returned as real image content blocks.
3+
The Scrapling MCP server exposes eleven tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), persistent browser session management, and page screenshots returned as real image content blocks. Fetch tools come in two modes: one-shot tools (`fetch`, `bulk_fetch`, `stealthy_fetch`, `bulk_stealthy_fetch`) each launch and close their own browser, while `session_fetch` fetches through a session opened with `open_session`.
44

55
All scraping tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str). The `screenshot` tool returns a list of MCP content blocks: an `ImageContent` (the screenshot bytes) followed by a `TextContent` (the post-redirect URL).
66

7-
## Tools
7+
## One-shot tools
88

99
### `get` -- HTTP request (single URL)
1010

@@ -66,11 +66,12 @@ Opens a Chromium browser via Playwright to render JavaScript. Suitable for dynam
6666
| `cookies` | list or null | null | Playwright-format cookies |
6767
| `timezone_id` | str or null | null | Browser timezone, e.g. `"America/New_York"` |
6868
| `locale` | str or null | null | Browser locale, e.g. `"en-GB"` |
69-
| `session_id` | str or null | null | Reuse a persistent session from `open_session` instead of creating a new browser |
69+
70+
This is a one-shot tool: it always launches its own browser. To fetch through a persistent session, use `session_fetch`.
7071

7172
### `bulk_fetch` -- Browser fetch (multiple URLs)
7273

73-
Concurrent browser version of `fetch`. Same parameters (including `session_id`) except `url` is replaced by `urls` (list of strings). Each URL opens in a separate browser tab. Returns a list of `ResponseModel`.
74+
Concurrent browser version of `fetch`. Same parameters except `url` is replaced by `urls` (list of strings). Each URL opens in a separate browser tab. Returns a list of `ResponseModel`.
7475

7576
### `stealthy_fetch` -- Stealth browser fetch (single URL)
7677

@@ -85,17 +86,18 @@ Anti-bot bypass fetcher with fingerprint spoofing. Use this for sites with Cloud
8586
| `block_webrtc` | bool | false | Force WebRTC to respect proxy settings (prevents IP leak) |
8687
| `allow_webgl` | bool | true | Keep WebGL enabled (disabling is detectable by WAFs) |
8788
| `additional_args` | dict or null | null | Extra Playwright context args (overrides Scrapling defaults) |
88-
| `session_id` | str or null | null | Reuse a persistent stealthy session from `open_session` |
8989

90-
All parameters from `fetch` are also accepted.
90+
All parameters from `fetch` are also accepted. Like `fetch`, this is a one-shot tool that launches its own browser; use `session_fetch` for a stealthy session.
9191

9292
### `bulk_stealthy_fetch` -- Stealth browser fetch (multiple URLs)
9393

94-
Concurrent stealth version. Same parameters (including `session_id`) as `stealthy_fetch` except `url` is replaced by `urls` (list of strings). Returns a list of `ResponseModel`.
94+
Concurrent stealth version. Same parameters as `stealthy_fetch` except `url` is replaced by `urls` (list of strings). Returns a list of `ResponseModel`.
95+
96+
## Session tools
9597

9698
### `open_session` -- Create a persistent browser session
9799

98-
Opens a browser session that stays alive across multiple fetch calls, avoiding the overhead of launching a new browser each time. Returns a `SessionCreatedModel` with `session_id`, `session_type`, `created_at`, `is_alive`, and `message`.
100+
Opens a browser session that stays alive across multiple `session_fetch` calls, avoiding the overhead of launching a new browser each time. It holds the browser-level configuration only; per-request options are passed to `session_fetch`. Returns a `SessionCreatedModel` with `session_id`, `session_type`, `created_at`, `is_alive`, `settings` (the session's effective configuration for the AI agent; empty for CDP sessions), and `message`.
99101

100102
**Key parameters:**
101103

@@ -104,17 +106,37 @@ Opens a browser session that stays alive across multiple fetch calls, avoiding t
104106
| `session_type` | `"dynamic"` / `"stealthy"` | required | Type of browser session to create |
105107
| `session_id` | str or null | null | Custom ID for the session. If omitted, a random 12-char hex ID is generated. Raises if already in use |
106108
| `headless` | bool | true | Run browser hidden or visible |
107-
| `max_pages` | int | 5 | Max concurrent browser tabs (1-50) |
108-
| `proxy` | str or dict or null | null | Proxy for all requests in this session |
109-
| `timeout` | number | 30000 | Default timeout in ms |
110-
| `solve_cloudflare` | bool | false | (Stealthy only) Auto-solve Cloudflare challenges |
111109
| `hide_canvas` | bool | false | (Stealthy only) Canvas fingerprint noise |
112110
| `block_webrtc` | bool | false | (Stealthy only) Block WebRTC IP leak |
113111
| `allow_webgl` | bool | true | (Stealthy only) Keep WebGL enabled |
114112

115-
Plus all other browser session parameters (`google_search`, `real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `extra_headers`, `cookies`, `disable_resources`, `network_idle`, `wait_selector`, `wait_selector_state`).
113+
Plus the other browser-level session parameters (`real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `proxy`, `solve_cloudflare`) are not set here; pass them to `session_fetch`.
114+
115+
One `session_fetch` works with either session type; `solve_cloudflare` only applies to a stealthy session.
116+
117+
### `session_fetch` -- Fetch through an open session (single URL)
118+
119+
Fetches one URL through a session opened with `open_session` (dynamic or stealthy). The session holds the browser-level configuration; every parameter here applies to this request only.
116120

117-
A dynamic session can only be used with `fetch`/`bulk_fetch`. A stealthy session can only be used with `stealthy_fetch`/`bulk_stealthy_fetch`.
121+
| Parameter | Type | Default | Description |
122+
|-----------------------|---------------------|--------------|---------------------------------------------------------------------------------|
123+
| `url` | str | required | URL to fetch |
124+
| `session_id` | str | required | ID of an open session created with `open_session` |
125+
| `extraction_type` | str | `"markdown"` | `"markdown"` / `"html"` / `"text"` |
126+
| `css_selector` | str or null | null | Narrow content before extraction |
127+
| `main_content_only` | bool | true | Restrict to `<body>` |
128+
| `wait` | number | 0 | Extra wait (ms) after page load before extraction |
129+
| `timeout` | number | 30000 | Timeout in **milliseconds** |
130+
| `google_search` | bool | true | Set a Google referer header |
131+
| `network_idle` | bool | false | Wait until no network activity for 500ms |
132+
| `load_dom` | bool | true | Wait for the page's JavaScript to fully load and execute |
133+
| `disable_resources` | bool | false | Block fonts, images, media, stylesheets, etc. for speed |
134+
| `wait_selector` | str or null | null | CSS selector to wait for before extraction |
135+
| `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` |
136+
| `extra_headers` | dict or null | null | Additional request headers |
137+
| `blocked_domains` | list or null | null | Domain names to block for this request (subdomains matched too) |
138+
| `proxy` | str or dict or null | null | Proxy for this request |
139+
| `solve_cloudflare` | bool | false | (Stealthy sessions only) Auto-solve Cloudflare challenges; errors on a dynamic session |
118140

119141
### `close_session` -- Close a persistent browser session
120142

@@ -128,7 +150,7 @@ Returns a `SessionClosedModel` with `session_id` and `message`.
128150

129151
### `list_sessions` -- List active sessions
130152

131-
Returns a list of `SessionInfo` objects, each with `session_id`, `session_type`, `created_at`, and `is_alive`.
153+
Returns a list of `SessionInfo` objects, each with `session_id`, `session_type`, `created_at`, `is_alive`, and `settings` (same as `open_session` returns).
132154

133155
No parameters.
134156

@@ -161,7 +183,7 @@ Requires an open browser session. Call `open_session` first, then pass the `sess
161183
| Multiple JS-rendered pages | `bulk_fetch` |
162184
| Cloudflare or strong anti-bot protection | `stealthy_fetch` (with `solve_cloudflare=true` for Turnstile) |
163185
| Multiple protected pages | `bulk_stealthy_fetch` |
164-
| Multiple pages from the same site | `open_session` + `fetch`/`stealthy_fetch` with `session_id` |
186+
| Multiple pages from the same site | `open_session` + `session_fetch` per page |
165187
| Need a screenshot of a page | `open_session` + `screenshot` with `session_id` |
166188

167189
Start with `get` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
@@ -253,7 +275,7 @@ The MCP server name when registering with a client is `ScraplingServer`. The com
253275

254276
## Connecting to remote browsers
255277

256-
`open_session` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with the fetch and screenshot tools as usual.
278+
`open_session` doesn't have to launch a browser locally. Pass a `cdp_url` and it connects to an already-running browser through the Chrome DevTools Protocol, whether that browser is on the same machine, another host, or a managed browser provider. Both session types (`dynamic` and `stealthy`) accept it, and the `session_id` you get back is used with `session_fetch` and `screenshot` as usual.
257279

258280
The URL can be a WebSocket endpoint (`ws://`/`wss://`), which is what managed browser providers hand out, or the HTTP endpoint of a browser started with `--remote-debugging-port=9222`, reached as `cdp_url="http://localhost:9222"`.
259281

0 commit comments

Comments
 (0)