Skip to content

Commit 1181fc6

Browse files
D4VinciD4Vinci
committed
feat(mcp)!: rename get to make_request and add more methods.
1 parent 93ae0ac commit 1181fc6

5 files changed

Lines changed: 83 additions & 39 deletions

File tree

‎agent-skill/Scrapling-Skill.zip‎

161 Bytes
Binary file not shown.

‎agent-skill/Scrapling-Skill/references/mcp-server.md‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -6,15 +6,18 @@ All scraping tools return a `ResponseModel` with fields: `status` (int), `conten
66

77
## One-shot tools
88

9-
### `get` -- HTTP request (single URL)
9+
### `make_request` -- HTTP request, any method (single URL)
1010

11-
Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable for static pages with no/low bot protection.
11+
Fast HTTP request with browser fingerprint impersonation (TLS, headers). Supports GET (default), POST, PUT, and DELETE via the `method` parameter. Suitable for static pages with no/low bot protection.
1212

1313
**Key parameters:**
1414

1515
| Parameter | Type | Default | Description |
1616
|---------------------|------------------------------------|--------------|--------------------------------------------------------------------|
1717
| `url` | str | required | URL to fetch |
18+
| `method` | `"GET"` / `"POST"` / `"PUT"` / `"DELETE"` | `"GET"` | HTTP method |
19+
| `data` | dict or str or null | null | Request body (form data). POST/PUT/DELETE only |
20+
| `json` | dict or list or null | null | Request body (JSON). POST/PUT/DELETE only |
1821
| `extraction_type` | `"markdown"` / `"html"` / `"text"` | `"markdown"` | Output format |
1922
| `css_selector` | str or null | null | CSS selector to narrow content (applied after `main_content_only`) |
2023
| `main_content_only` | bool | true | Restrict to `<body>` content |
@@ -34,9 +37,9 @@ Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable fo
3437
| `params` | dict or null | null | Query string parameters |
3538
| `verify` | bool | true | Verify HTTPS certificates |
3639

37-
### `bulk_get` -- HTTP request (multiple URLs)
40+
### `bulk_get` -- HTTP GET request (multiple URLs)
3841

39-
Async concurrent version of `get`. Same parameters except `url` is replaced by `urls` (list of strings). All URLs are fetched in parallel. Returns a list of `ResponseModel`.
42+
Async concurrent GET-only version of `make_request`. Same parameters except `url` is replaced by `urls` (list of strings) and there are no `method`/`data`/`json` parameters. All URLs are fetched in parallel. Returns a list of `ResponseModel`.
4043

4144
### `fetch` -- Browser fetch (single URL)
4245

@@ -110,7 +113,7 @@ Opens a browser session that stays alive across multiple `session_fetch` calls,
110113
| `block_webrtc` | bool | false | (Stealthy only) Block WebRTC IP leak |
111114
| `allow_webgl` | bool | true | (Stealthy only) Keep WebGL enabled |
112115

113-
Plus the other browser-level session parameters (`real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `proxy`, `solve_cloudflare`) are not set here; pass them to `session_fetch`.
116+
Plus the other browser-level session parameters (`proxy`, `real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `cookies`, `executable_path`, `additional_args`). Per-request options (`timeout`, `wait`, `google_search`, `network_idle`, `disable_resources`, `wait_selector`, `wait_selector_state`, `extra_headers`, `solve_cloudflare`) are not set here; pass them to `session_fetch`.
114117

115118
One `session_fetch` works with either session type; `solve_cloudflare` only applies to a stealthy session.
116119

@@ -135,7 +138,6 @@ Fetches one URL through a session opened with `open_session` (dynamic or stealth
135138
| `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` |
136139
| `extra_headers` | dict or null | null | Additional request headers |
137140
| `blocked_domains` | list or null | null | Domain names to block for this request (subdomains matched too) |
138-
| `proxy` | str or dict or null | null | Proxy for this request |
139141
| `solve_cloudflare` | bool | false | (Stealthy sessions only) Auto-solve Cloudflare challenges; errors on a dynamic session |
140142

141143
### `close_session` -- Close a persistent browser session
@@ -177,16 +179,16 @@ Requires an open browser session. Call `open_session` first, then pass the `sess
177179

178180
| Scenario | Tool |
179181
|------------------------------------------|---------------------------------------------------------------|
180-
| Static page, no bot protection | `get` |
181-
| Multiple static pages | `bulk_get` |
182+
| Static page, no bot protection | `make_request` |
183+
| Multiple static pages | `bulk_get` |
182184
| JavaScript-rendered / SPA page | `fetch` |
183185
| Multiple JS-rendered pages | `bulk_fetch` |
184186
| Cloudflare or strong anti-bot protection | `stealthy_fetch` (with `solve_cloudflare=true` for Turnstile) |
185187
| Multiple protected pages | `bulk_stealthy_fetch` |
186188
| Multiple pages from the same site | `open_session` + `session_fetch` per page |
187189
| Need a screenshot of a page | `open_session` + `screenshot` with `session_id` |
188190

189-
Start with `get` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
191+
Start with `make_request` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
190192

191193
## Content extraction tips
192194

‎docs/ai/mcp-server.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ The Scrapling MCP Server provides eleven powerful tools for web scraping, split
1111
### One-shot tools
1212

1313
#### 🚀 Basic HTTP Scraping
14-
- **`get`**: Fast HTTP requests with browser fingerprint impersonation, generating real browser headers matching the TLS version, HTTP/3, and more!
15-
- **`bulk_get`**: An async version of the above tool that allows scraping of multiple URLs at the same time!
14+
- **`make_request`**: Fast HTTP requests with any method (GET, POST, PUT, DELETE) and browser fingerprint impersonation, generating real browser headers matching the TLS version, HTTP/3, and more!
15+
- **`bulk_get`**: An async GET-only version of the above tool that allows scraping of multiple URLs at the same time!
1616

1717
#### 🌐 Dynamic Content Scraping
1818
- **`fetch`**: Rapidly fetch dynamic content with Chromium/Chrome browser with complete control over the request/browser, and more!
@@ -290,7 +290,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
290290
Scrape the main content from https://example.com and convert it to markdown format.
291291
```
292292
293-
Claude will use the `get` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for 3 attempts, unless you instruct it otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt.
293+
Claude will use the `make_request` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for 3 attempts, unless you instruct it otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt.
294294
295295
A more optimized version of the same prompt would be:
296296
```
@@ -404,7 +404,7 @@ And so on, you get the idea. Your creativity is the key here.
404404
Here is some technical advice for you.
405405
406406
### 1. Choose the Right Tool
407-
- **`get`**: Fast, simple websites
407+
- **`make_request`**: Fast, simple websites
408408
- **`fetch`**: Sites with JavaScript/dynamic content
409409
- **`stealthy_fetch`**: Protected sites, Cloudflare, anti-bot systems
410410

‎scrapling/core/ai.py‎

Lines changed: 31 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
extraction_types,
4141
SelectorWaitStates,
4242
FollowRedirects,
43+
SUPPORTED_HTTP_METHODS,
4344
)
4445

4546
SessionType = Literal["dynamic", "stealthy"]
@@ -378,13 +379,16 @@ async def _capture(page: Any) -> None:
378379
return [image, TextContent(type="text", text=captured["url"])]
379380

380381
@staticmethod
381-
async def get(
382+
async def make_request(
382383
url: str,
384+
method: SUPPORTED_HTTP_METHODS = "GET",
383385
impersonate: ImpersonateType = "chrome",
384386
extraction_type: extraction_types = "markdown",
385387
css_selector: Optional[str] = None,
386388
main_content_only: bool = True,
387389
params: Optional[Dict] = None,
390+
data: Optional[Dict[str, str] | str] = None,
391+
json: Optional[Dict | List] = None,
388392
headers: Optional[Mapping[str, Optional[str]]] = None,
389393
cookies: Optional[Dict[str, str]] = None,
390394
timeout: Optional[int | float] = 30,
@@ -399,15 +403,18 @@ async def get(
399403
http3: Optional[bool] = False,
400404
stealthy_headers: Optional[bool] = True,
401405
) -> ResponseModel:
402-
"""Make GET HTTP request to a URL and return a structured output of the result.
406+
"""Make an HTTP request to a URL with any method (GET, POST, PUT, DELETE) and return a structured output of the result.
403407
Only suitable for low-mid protection levels.
404408
405409
:param url: The URL to request.
410+
:param method: The HTTP method to use: "GET" (default), "POST", "PUT", or "DELETE".
406411
:param impersonate: Browser version to impersonate its fingerprint. It's using the latest chrome version by default.
407412
:param extraction_type: The type of content to extract from the page: "markdown", "html", or "text".
408413
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page.
409414
:param main_content_only: Whether to extract only the main content of the page. The main content here is the data inside the `<body>` tag.
410415
:param params: Query string parameters for the request.
416+
:param data: Form data for the request body. Used with "POST", "PUT", and "DELETE" only.
417+
:param json: A JSON-serializable object for the request body. Used with "POST", "PUT", and "DELETE" only.
411418
:param headers: Headers to include in the request.
412419
:param cookies: Cookies to use in the request.
413420
:param timeout: Number of seconds to wait before timing out.
@@ -424,28 +431,32 @@ async def get(
424431
:param http3: Whether to use HTTP3. It might be problematic if used it with `impersonate`.
425432
:param stealthy_headers: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.
426433
"""
427-
results = await ScraplingMCPServer.bulk_get(
428-
urls=[url],
429-
impersonate=impersonate,
430-
extraction_type=extraction_type,
431-
css_selector=css_selector,
432-
main_content_only=main_content_only,
434+
normalized_proxy_auth = _normalize_credentials(proxy_auth)
435+
normalized_auth = _normalize_credentials(auth)
436+
437+
request_kwargs: Dict[str, Any] = dict(
438+
auth=normalized_auth,
439+
proxy=proxy,
440+
http3=http3,
441+
verify=verify,
433442
params=params,
434443
headers=headers,
435444
cookies=cookies,
436445
timeout=timeout,
437-
follow_redirects=follow_redirects,
438-
max_redirects=max_redirects,
439446
retries=retries,
447+
proxy_auth=normalized_proxy_auth,
440448
retry_delay=retry_delay,
441-
proxy=proxy,
442-
proxy_auth=proxy_auth,
443-
auth=auth,
444-
verify=verify,
445-
http3=http3,
449+
impersonate=impersonate,
450+
max_redirects=max_redirects,
451+
follow_redirects=follow_redirects,
446452
stealthy_headers=stealthy_headers,
447453
)
448-
return results[0]
454+
if method != "GET":
455+
request_kwargs.update(data=data, json=json)
456+
457+
async with FetcherSession() as session:
458+
page = await getattr(session, method.lower())(url, **request_kwargs)
459+
return _translate_response(page, extraction_type, css_selector, main_content_only)
449460

450461
@staticmethod
451462
async def bulk_get(
@@ -950,7 +961,7 @@ def _build_server(self, host: str, port: int) -> MCPServer:
950961
"cache_hints": {"tools/list": CacheHint(ttl_ms=3_600_000, scope="public")},
951962
"instructions": """Follow these instructions precisely:
952963
1. When the `open_session` tool is used, make sure to close the session with `close_session` after you finish, and use `list_sessions` if you lose track of the open sessions or their effective settings.
953-
2. If the user didn't specify which tool to use, start with the `get` tool, then escalate. The `get` tool and the bulk version are suitable only for low-to-mid protection levels.
964+
2. If the user didn't specify which tool to use, start with the `make_request` tool (a plain HTTP request, defaulting to GET; set `method` for POST/PUT/DELETE), then escalate. The `make_request` tool and `bulk_get` (its GET-only bulk version) are suitable only for low-to-mid protection levels.
954965
For high-protection levels or websites that require JS loading, use the other tools directly.
955966
3. For all tools, if the `css_selector` resolves to more than one element, all the elements will be returned.
956967
4. For all fetch tools, the `extraction_type` parameter controls the format of the returned content: "markdown" (default) converts the page content to Markdown, "html" returns the raw HTML, and "text" returns the text content of the page.
@@ -981,9 +992,9 @@ def _build_server(self, host: str, port: int) -> MCPServer:
981992
)
982993
# HTTP tools
983994
server.add_tool(
984-
self.get,
985-
title="get",
986-
description=self.get.__doc__,
995+
self.make_request,
996+
title="make_request",
997+
description=self.make_request.__doc__,
987998
structured_output=True,
988999
annotations=_FETCH_TOOL_ANNOTATIONS,
9891000
)

‎tests/ai/test_ai_mcp.py‎

Lines changed: 37 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@
3434

3535

3636
def test_translate_response_strips_control_characters():
37-
"""Pages with control chars like U+0008 must not crash the get/fetch path (issue #366)"""
37+
"""Pages with control chars like U+0008 must not crash the request/fetch path (issue #366)"""
3838
html = "<html><body><p>Hello\x08World</p>\t\n<div>Foo\x0cbar</div></body></html>"
3939
page = Response(
4040
url="https://jfinal.com/doc/1-5",
@@ -126,13 +126,22 @@ def server(self):
126126
return ScraplingMCPServer()
127127

128128
@pytest.mark.asyncio
129-
async def test_get_tool(self, server, test_url):
130-
"""Test the get tool method"""
131-
result = await server.get(url=test_url, extraction_type="markdown")
129+
async def test_make_request_tool(self, server, test_url):
130+
"""Test the make_request tool method with a default GET"""
131+
result = await server.make_request(url=test_url, extraction_type="markdown")
132132
assert isinstance(result, ResponseModel)
133133
assert result.status == 200
134134
assert result.url == test_url
135135

136+
@pytest.mark.asyncio
137+
async def test_make_request_post_tool(self, server, httpbin):
138+
"""Test the make_request tool method with a POST body"""
139+
result = await server.make_request(
140+
url=f"{httpbin.url}/post", method="POST", json={"key": "value"}, extraction_type="text"
141+
)
142+
assert isinstance(result, ResponseModel)
143+
assert result.status == 200
144+
136145
@pytest.mark.asyncio
137146
async def test_bulk_get_tool(self, server, test_url):
138147
"""Test the bulk_get tool method"""
@@ -422,9 +431,9 @@ async def test_dynamic_session_fetch_forwards_the_per_request_params(self, monke
422431
"wait_selector_state": "attached",
423432
"extra_headers": None,
424433
"blocked_domains": None,
425-
"proxy": None,
426434
}, forwarded
427435
assert "solve_cloudflare" not in forwarded, "solve_cloudflare must not reach a dynamic session"
436+
assert "proxy" not in forwarded, "proxy is session-level (open_session), never forwarded per request"
428437

429438
@pytest.mark.asyncio
430439
async def test_stealthy_session_fetch_forwards_solve_cloudflare(self, monkeypatch):
@@ -515,6 +524,23 @@ def test_open_session_holds_no_per_request_params(self):
515524
f"open_session still carries per-request params: {params & set(_STEALTH_FETCH_KEYS)}"
516525
)
517526

527+
def test_proxy_is_session_level_not_per_request(self):
528+
"""A session runs one tab, so proxy is set once on open_session, never per request"""
529+
assert "proxy" in inspect.signature(ScraplingMCPServer.open_session).parameters
530+
assert "proxy" not in inspect.signature(ScraplingMCPServer.session_fetch).parameters
531+
assert "proxy" not in _STEALTH_FETCH_KEYS
532+
533+
@pytest.mark.asyncio
534+
async def test_open_session_forwards_proxy_to_the_session(self, monkeypatch):
535+
"""The session-level proxy reaches the underlying session so it applies to every fetch"""
536+
monkeypatch.setattr("scrapling.core.ai.AsyncDynamicSession", _FakeDynamicSession)
537+
_FakeDynamicSession.instances = []
538+
server = ScraplingMCPServer()
539+
540+
await server.open_session(session_type="dynamic", proxy="http://user:pass@host:8080")
541+
542+
assert _FakeDynamicSession.instances[0].kwargs["proxy"] == "http://user:pass@host:8080"
543+
518544

519545
class TestSessionSettingsReceipt:
520546
"""open_session and list_sessions return the session's effective settings."""
@@ -822,6 +848,11 @@ async def test_fetch_tools_expose_real_defaults_and_no_session_id(self):
822848
assert props["timeout"]["default"] == 30000, f"{name} hides the real timeout default"
823849
assert props["google_search"]["default"] is True
824850

851+
request_props = tools["make_request"].input_schema["properties"]
852+
assert request_props["method"]["default"] == "GET"
853+
assert "data" in request_props and "json" in request_props
854+
assert "method" not in tools["bulk_get"].input_schema["properties"]
855+
825856
session_props = tools["session_fetch"].input_schema["properties"]
826857
assert session_props["timeout"]["default"] == 30000
827858
assert "solve_cloudflare" in session_props
@@ -849,7 +880,7 @@ async def test_server_metadata_and_tool_annotations(self):
849880
annotations = {tool.name: tool.annotations for tool in result.tools if tool.annotations is not None}
850881
assert len(annotations) == 11
851882
for name in (
852-
"get",
883+
"make_request",
853884
"bulk_get",
854885
"fetch",
855886
"bulk_fetch",

0 commit comments

Comments
 (0)