Merge branch 'dev' into fix/static-zero-retries-skips-request

Author: D4Vinci

CHANGELOG.md

@@ -23,8 +23,8 @@
     <a href="https://hub.docker.com/r/pyd4vinci/scrapling" target="_blank">
         <img alt="Docker Pulls" src="https://img.shields.io/docker/pulls/pyd4vinci/scrapling?labelColor=%20%23FDB062&logo=Docker&labelColor=%20%23528bff"></a>
     <a href="https://clickpy.clickhouse.com/dashboard/scrapling" rel="nofollow"><img src="https://img.shields.io/pypi/dm/scrapling" alt="PyPI package downloads"></a>
-    <a href="https://github.com/D4Vinci/Scrapling/tree/main/agent-skill" alt="AI Agent Skill directory">
-        <img alt="Static Badge" src="https://img.shields.io/badge/Skill-black?style=flat&label=Agent&link=https%3A%2F%2Fgithub.qkg1.top%2FD4Vinci%2FScrapling%2Ftree%2Fmain%2Fagent-skill"></a>
+    <a href="https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html" alt="AI Agent Skill">
+        <img alt="Static Badge" src="https://img.shields.io/badge/Skill-black?style=flat&label=Agent&link=https%3A%2F%2Fscrapling.readthedocs.io%2Fen%2Flatest%2Fai%2Fagent-skill.html"></a>
     <a href="https://clawhub.ai/D4Vinci/scrapling-official" alt="OpenClaw Skill">
         <img alt="OpenClaw Skill" src="https://img.shields.io/badge/Clawhub-darkred?style=flat&label=OpenClaw&link=https%3A%2F%2Fclawhub.ai%2FD4Vinci%2Fscrapling-official"></a>
     <br/>
@@ -55,7 +55,7 @@
 
 Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.
 
-Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python. One library, zero compromises.
+Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume, automatic proxy rotation, and a crawl speed that adapts to how fast each website responds and backs off when it starts blocking you - all in a few lines of Python. One library, zero compromises.
 
 Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
 
@@ -91,12 +91,12 @@ MySpider().start()
 <table>
   <tr>
     <td width="200">
-      <a href="https://go.nodemaven.com/scraplingjuly" target="_blank" title="Proxies with the Highest IP Scores">
+      <a href="https://go.nodemaven.com/scraplingaugust" target="_blank" title="Proxies with the Highest IP Scores">
         <img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/NodeMaven.jpg" width="240" height="100">
       </a>
     </td>
     <td>
-    <a href="https://go.nodemaven.com/scraplingjuly" target="_blank">NodeMaven</a> - reliable proxy provider with the highest quality IP on the market. Use promo code SCRAPLING35 for 35% discount on proxies.
+    <a href="https://go.nodemaven.com/scraplingaugust" target="_blank">NodeMaven</a> - The most efficient proxy provider for Web Scraping and Automation with the Highest Quality IP on the market. Use code SCRAPLING35 for 35% discount.
     </td>
   </tr>
   <tr>
@@ -221,6 +221,7 @@ MySpider().start()
 <!-- sponsors -->
 
 <a href="https://www.novada.com/?d4vinci-scrapling" target="_blank" title="The All-in-One Solution for Every Data Scraping Scenario"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/novada.jpg"></a>
+<a href="https://cloro.dev/?utm_source=referral&utm_medium=scrapling" target="_blank" title="The search API for the AI era"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/cloro.jpg"></a>
 
 <br/>
 
@@ -265,12 +266,15 @@ MySpider().start()
 - **Background API Capture**: Pass a URL pattern to `capture_xhr`, and all matching XHR/fetch responses the page makes while loading are collected for you as `Response` objects in `response.captured_xhr` - grab a site's API data without reverse-engineering the requests yourself.
 - **Async Support**: Complete async support across all fetchers and dedicated async session classes.
 
-### Adaptive Scraping & AI Integration
+### Adaptive Scraping
 - 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.
 - 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
 - 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
-- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features powerful, custom capabilities that leverage Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE)) It can also keep browser sessions open across calls, take page screenshots, and drive remote browsers over CDP.
-- 🧠 **Agent Skill**: A ready-to-install [Agent Skill](https://github.com/D4Vinci/Scrapling/tree/main/agent-skill) that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
+
+### AI Features
+- 🤖 **MCP Server**: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with one-shot or session-based tools covering plain HTTP requests (any method), browser fetches, and stealth fetches that bypass Cloudflare. Pages are narrowed with CSS selectors and stripped of prompt-injection content before the AI sees them, so the agent reads less, costs less, and can't be hijacked by hidden text. Screenshots, remote browsers over CDP, and a secure-by-default HTTP transport are included. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
+- 🧠 **Agent Skill**: A ready-to-install [Agent Skill](https://scrapling.readthedocs.io/en/latest/ai/agent-skill.html) that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
+- 📚 **RAG-ready Markdown**: Turn any page into clean, sanitized, LLM-ready Markdown with one line (`page.markdown()`), or crawl a whole website into a Markdown corpus with the `SiteToMarkdownSpider` template, all without an LLM in the loop. ([docs](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html))
 
 ### High-Performance & battle-tested Architecture
 - 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
@@ -540,6 +544,10 @@ pip install scrapling
        ```bash
        pip install "scrapling[ai]"
        ```
+   - Install dependencies for ([building RAG systems](https://scrapling.readthedocs.io/en/latest/ai/building-rag-systems.html)):
+       ```bash
+       pip install "scrapling[rag]"
+       ```
    - Install shell features (Web Scraping shell and the `extract` command): 
        ```bash
        pip install "scrapling[shell]"

README.md

@@ -398,6 +398,7 @@ async with AsyncDynamicSession(capture_xhr=r"https://api\.example\.com/.*") as s
 ## References
 You already had a good glimpse of what the library can do. Use the references below to dig deeper when needed
 - `references/mcp-server.md` - MCP server tools, persistent session management, remote browsers over CDP, authentication, and capabilities
+- `references/building-rag-systems.md` - Converting pages/websites to LLM-ready Markdown with `Response.markdown()` and `SiteToMarkdownSpider` for RAG pipelines
 - `references/parsing` - Everything you need for parsing HTML
 - `references/fetching` - Everything you need to fetch websites and session persistence
 - `references/spiders` - Everything you need to write spiders, proxy rotation, and advanced features. It follows a Scrapy-like format

agent-skill/Scrapling-Skill.zip

@@ -0,0 +1,104 @@
+# Building RAG Systems
+
+RAG pipelines are only as good as the text you feed them. Raw HTML wastes tokens on markup, navigation, scripts, and tracking noise, and it can even carry hidden prompt-injection content straight into your LLM. Scrapling turns pages and whole websites into clean, sanitized Markdown with no LLM in the loop, so your ingestion runs fast and costs nothing per page.
+
+## Installation
+
+```bash
+pip install "scrapling[rag]"
+
+scrapling install
+```
+
+The `rag` extra installs the fetchers with Markdown conversion support (the `ai`, `shell`, and `all` extras include it too). The `scrapling install` command downloads the browser dependencies, which you only need for the browser-based fetchers.
+
+## One page to Markdown
+
+Every [Response](fetching/choosing.md) has a `markdown()` method:
+
+```python
+from scrapling.fetchers import Fetcher
+
+markdown = Fetcher.get("https://example.com").markdown(main_content_only=True)
+```
+
+It works with all fetchers, so pages behind Cloudflare are one line away too:
+
+```python
+from scrapling.fetchers import StealthyFetcher
+
+markdown = StealthyFetcher.fetch("https://protected.example.com", solve_cloudflare=True).markdown(main_content_only=True)
+```
+
+Two arguments control the output:
+
+- `main_content_only`: Convert only the content inside the page's `<body>` tag.
+- `css_selector`: Convert only the elements matching a CSS selector (all matches are concatenated). Use it to extract exactly the part your pipeline needs and save tokens:
+
+```python
+markdown = Fetcher.get("https://example.com/docs/page").markdown(css_selector="article")
+```
+
+Whatever you pass, scripts, styles, and hidden content are always removed before conversion. This is the same cleaning the [MCP server](mcp-server.md) uses to protect AI agents from prompt injection: CSS-hidden elements, `aria-hidden` elements, `<template>` tags, HTML comments, and zero-width characters never reach your model.
+
+## A whole website to a Markdown corpus
+
+The `SiteToMarkdownSpider` template crawls a website and converts every page, powered by the [spiders framework](spiders/architecture.md), so you get concurrency, autothrottle, robots.txt compliance, and pause/resume for free:
+
+```python
+from scrapling.spiders import SiteToMarkdownSpider
+
+class DocsSpider(SiteToMarkdownSpider):
+    name = "docs"
+    start_urls = ["https://example.com/docs/"]
+    allowed_domains = {"example.com"}
+    output_dir = "docs_markdown"
+    max_pages = 200
+
+result = DocsSpider().start()
+result.items.to_jsonl("docs.jsonl")
+```
+
+Each crawled page becomes one item with `url`, `title`, and `markdown` keys. With `output_dir` set, each page is also written to a Markdown file named after its URL, so the run above gives you both a folder of `.md` files and a `docs.jsonl` ready for ingestion.
+
+The template requires `allowed_domains` so the crawl stays bound to the target website. The options:
+
+- `css_selector` / `main_content_only`: Passed to `markdown()` for every page, with `main_content_only` enabled by default.
+- `output_dir`: When set, writes one Markdown file per page.
+- `max_pages`: Maximum number of pages to convert. Requests already queued when the cap hits may still be fetched, but they aren't converted. `0` (the default) disables it.
+
+Every page link inside `allowed_domains` is followed by default. Since the template builds on [CrawlSpider](spiders/generic-templates.md), override `rules()` with your own [LinkExtractor](spiders/generic-templates.md) to control the crawl: `allow` narrows it to the URL patterns you want, and `deny` drops the patterns you don't (login pages, tag listings, print views, etc.):
+
+```python
+from scrapling.spiders import CrawlRule, LinkExtractor, SiteToMarkdownSpider
+
+class DocsSpider(SiteToMarkdownSpider):
+    name = "docs"
+    start_urls = ["https://example.com/"]
+    allowed_domains = {"example.com"}
+
+    def rules(self):
+        return [CrawlRule(LinkExtractor(allow=r"/docs/", deny=[r"/docs/changelog/", r"\?print="]))]
+```
+
+`deny` wins over `allow`, and a rule can carry a `priority` or a `process_request` hook as with any **CrawlSpider**.
+
+## Feeding a vector store
+
+The JSONL output plugs into any embedding pipeline. A minimal example:
+
+```python
+import json
+
+with open("docs.jsonl") as f:
+    for line in f:
+        page = json.loads(line)
+        for chunk in split_into_chunks(page["markdown"]):
+            vector_store.add(text=chunk, metadata={"url": page["url"], "title": page["title"]})
+```
+
+Use `css_selector` on the spider to cut boilerplate before chunking instead of cleaning it downstream. The less noise you embed, the better your retrieval.
+
+## Interactive alternatives
+
+For conversational scraping instead of pipelines, the [MCP server](mcp-server.md) gives your AI chatbot the same Markdown extraction as tools, and this agent skill teaches coding agents to write this code themselves.

agent-skill/Scrapling-Skill/SKILL.md

@@ -75,4 +75,10 @@ page.captured_xhr    # List of captured XHR/fetch responses (when capture_xhr is
 ```
 All fetchers return the `Response` object.
 
+The `Response` object can also convert the page to clean, LLM-ready Markdown in one line:
+```python
+markdown = Fetcher.get('https://example.com').markdown(main_content_only=True)
+```
+Scripts, styles, and hidden/prompt-injection content are always removed before conversion, and you can pass `css_selector` to convert specific elements only. It requires the `rag` extra (included in `ai`/`shell`/`all` too). See `../building-rag-systems.md` for the full guide.
+
 **Note:** Unlike the [Selector](parsing/main_classes.md#selector) class, the `Response` class's body is always bytes since v0.4.

agent-skill/Scrapling-Skill/references/building-rag-systems.md

@@ -325,14 +325,19 @@ async def scrape_multiple_sites():
         return pages
 ```
 
-You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages that can be displayed at once. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
+You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
 
-1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
-2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
+2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
+3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+
+Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
 
 This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
 
-In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
+Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
+
+Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
 
 ### Session Benefits
 

agent-skill/Scrapling-Skill/references/fetching/choosing.md

@@ -228,14 +228,19 @@ async def scrape_multiple_sites():
         return pages
 ```
 
-You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages that can be displayed at once. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
+You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
 
-1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
-2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
+2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
+3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+
+Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
 
 This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
 
-In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
+Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
+
+Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
 
 ### Session Benefits