feat(fetchers): keep browser tabs open and reuse them across requests

Browser sessions no longer close a tab after its request is completed. The tab returns to the pool as ready, and the next request reuses it, reapplying its own timeouts, extra headers, and resource/domain routes (after `unroute_all`) so nothing leaks between requests. Tabs that hit an error or got closed by the browser are closed and evicted, and the internal response listener is detached after every request so reused tabs don't stack handlers.

Adds `close_pages()` to all browser sessions to close every open tab; the next request opens a fresh one. `PagePool` gains `get_ready_page`/`remove_page`/`clear` and `PageInfo.mark_ready`, new pages start busy, and the unused `cleanup_error_pages` is removed.

Proxy-rotation contexts keep closing per request. Docs and the agent skill describe the new lifecycle.

Author: D4Vinci

agent-skill/Scrapling-Skill.zip

@@ -325,14 +325,19 @@ async def scrape_multiple_sites():
         return pages
 ```
 
-You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages that can be displayed at once. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
+You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
 
-1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
-2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
+2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
+3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+
+Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
 
 This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
 
-In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
+Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
+
+Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
 
 ### Session Benefits
 

agent-skill/Scrapling-Skill/references/fetching/dynamic.md

@@ -228,14 +228,19 @@ async def scrape_multiple_sites():
         return pages
 ```
 
-You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages that can be displayed at once. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
+You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
 
-1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
-2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
+2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
+3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+
+Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
 
 This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
 
-In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
+Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
+
+Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
 
 ### Session Benefits
 

agent-skill/Scrapling-Skill/references/fetching/stealthy.md

@@ -151,14 +151,19 @@ async def scrape_multiple_sites():
         return pages
 ```
 
-You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages that can be displayed at once. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
+You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
 
-1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
-2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
+2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
+3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+
+Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
 
 This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
 
-In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
+Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
+
+Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
 
 ### Session Benefits
 

docs/fetching/dynamic.md

@@ -240,14 +240,19 @@ async def scrape_multiple_sites():
         return pages
 ```
 
-You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages that can be displayed at once. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
+You may have noticed the `max_pages` argument. It enables the fetcher to keep a **pool of Browser tabs**, and you set the maximum number of tabs that can be open at once. Tabs stay open after their request finishes, so with each request, the library will:
 
-1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
-2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+1. Reuse a free tab if there's one. Every request applies its own tab-level settings (`timeout`, `extra_headers`, `disable_resources`, `blocked_domains`, etc.) to the tab it gets, so nothing leaks from the previous request.
+2. Otherwise, open a new tab if the number of open tabs is lower than `max_pages`.
+3. Otherwise, keep checking every subsecond for a tab to become free for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
+
+Tabs that hit an error are closed and replaced, and you can close all the open tabs yourself at any point with `session.close_pages()`, then the next request opens a fresh one.
 
 This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
 
-In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
+Keeping the tabs open also means the page you fetched is still there for the next request, so a `page_setup` function on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
+
+Versions 0.3.2 to 0.4.14 closed every tab after its request because reusing tabs used to leak settings between requests. Since 0.4.15, the settings are reset on every reuse, so the tabs stay open.
 
 ### Session Benefits
 

docs/fetching/stealthy.md

@@ -1,7 +1,7 @@
 from time import time
 from re import search as re_search
 from asyncio import sleep as asyncio_sleep, Lock
-from contextlib import contextmanager, asynccontextmanager
+from contextlib import contextmanager, asynccontextmanager, suppress
 
 from playwright.sync_api._generated import Page
 from playwright.sync_api import (
@@ -63,11 +63,18 @@ def __init__(self, max_pages: int = 1):
     def start(self) -> None:
         pass
 
+    def close_pages(self) -> None:
+        """Close every open tab in the session's pool. The next request opens a fresh tab."""
+        for page_info in self.page_pool.clear():
+            with suppress(Exception):
+                page_info.page.close()
+
     def close(self):  # pragma: no cover
         """Close all resources"""
         if not self._is_alive:
             return
 
+        self.close_pages()
         if self.context:
             self.context.close()
             self.context = None
@@ -107,21 +114,21 @@ def _get_page(
         blocked_domains: Optional[Set[str]] = None,
         context: Optional[BrowserContext] = None,
     ) -> PageInfo[Page]:  # pragma: no cover
-        """Get a new page to use"""
-        # No need to check if a page is available or not in sync code because the code blocked before reaching here till the page closed, ofc.
-        ctx = context if context is not None else self.context
-        assert ctx is not None, "Browser context not initialized"
-        page = ctx.new_page()
+        """Get a ready page from the pool, or open a new one"""
+        page_info = self.page_pool.get_ready_page() if context is None else None
+        if page_info is None:
+            ctx = context if context is not None else self.context
+            assert ctx is not None, "Browser context not initialized"
+            page_info = self.page_pool.add_page(ctx.new_page())
+
+        page = cast(Page, page_info.page)
         page.set_default_navigation_timeout(timeout)
         page.set_default_timeout(timeout)
-        if extra_headers:
-            page.set_extra_http_headers(extra_headers)
-
+        page.set_extra_http_headers(extra_headers or {})
+        page.unroute_all(behavior="ignoreErrors")
         if disable_resources or blocked_domains:
             page.route("**/*", create_intercept_handler(disable_resources, blocked_domains))
-        page_info = self.page_pool.add_page(page)
-        page_info.mark_busy()
-        return page_info
+        return cast(PageInfo[Page], page_info)
 
     def get_pool_stats(self) -> Dict[str, int]:
         """Get statistics about the current page pool"""
@@ -202,17 +209,21 @@ def _page_generator(
                 page_info = self._get_page(timeout, extra_headers, disable_resources, blocked_domains, context=context)
                 yield page_info
             finally:
-                if page_info is not None and page_info in self.page_pool.pages:
-                    self.page_pool.pages.remove(page_info)
+                if page_info is not None:
+                    self.page_pool.remove_page(page_info)
                 context.close()
         else:
             # Standard mode: use PagePool with persistent context
             page_info = self._get_page(timeout, extra_headers, disable_resources, blocked_domains)
             try:
                 yield page_info
             finally:
-                page_info.page.close()
-                self.page_pool.pages.remove(page_info)
+                if page_info.state == "error" or page_info.page.is_closed():
+                    with suppress(Exception):
+                        page_info.page.close()
+                    self.page_pool.remove_page(page_info)
+                else:
+                    page_info.mark_ready()
 
 
 class AsyncSession:
@@ -234,11 +245,18 @@ def __init__(self, max_pages: int = 1):
     async def start(self) -> None:
         pass
 
+    async def close_pages(self) -> None:
+        """Close every open tab in the session's pool. The next request opens a fresh tab."""
+        for page_info in self.page_pool.clear():
+            with suppress(Exception):
+                await cast(AsyncPage, page_info.page).close()
+
     async def close(self):
         """Close all resources"""
         if not self._is_alive:  # pragma: no cover
             return
 
+        await self.close_pages()
         if self.context:
             await self.context.close()
             self.context = None  # pyright: ignore
@@ -280,35 +298,37 @@ async def _get_page(
         blocked_domains: Optional[Set[str]] = None,
         context: Optional[AsyncBrowserContext] = None,
     ) -> PageInfo[AsyncPage]:  # pragma: no cover
-        """Get a new page to use"""
+        """Get a ready page from the pool, or open a new one"""
         ctx = context if context is not None else self.context
         if TYPE_CHECKING:
             assert ctx is not None, "Browser context not initialized"
 
         async with self._lock:
-            # If we're at max capacity after cleanup, wait for busy pages to finish
-            if context is None and self.page_pool.pages_count >= self.max_pages:
-                # Only applies when using persistent context
+            page_info = self.page_pool.get_ready_page() if context is None else None
+            if page_info is None and context is None and self.page_pool.pages_count >= self.max_pages:
+                # At max capacity with the persistent context, so wait for a busy page to become ready
                 start_time = time()
                 while time() - start_time < self._max_wait_for_page:
                     await asyncio_sleep(0.05)
-                    if self.page_pool.pages_count < self.max_pages:
+                    page_info = self.page_pool.get_ready_page()
+                    if page_info is not None:
                         break
                 else:
                     raise TimeoutError(
                         f"No pages finished to clear place in the pool within the {self._max_wait_for_page}s timeout period"
                     )
 
-            page = await ctx.new_page()
+            if page_info is None:
+                page_info = self.page_pool.add_page(await ctx.new_page())
+
+            page = cast(AsyncPage, page_info.page)
             page.set_default_navigation_timeout(timeout)
             page.set_default_timeout(timeout)
-            if extra_headers:
-                await page.set_extra_http_headers(extra_headers)
-
+            await page.set_extra_http_headers(extra_headers or {})
+            await page.unroute_all(behavior="ignoreErrors")
             if disable_resources or blocked_domains:
                 await page.route("**/*", create_async_intercept_handler(disable_resources, blocked_domains))
-
-            return self.page_pool.add_page(page)
+            return cast(PageInfo[AsyncPage], page_info)
 
     def get_pool_stats(self) -> Dict[str, int]:
         """Get statistics about the current page pool"""
@@ -391,17 +411,21 @@ async def _page_generator(
                 )
                 yield page_info
             finally:
-                if page_info is not None and page_info in self.page_pool.pages:
-                    self.page_pool.pages.remove(page_info)
+                if page_info is not None:
+                    self.page_pool.remove_page(page_info)
                 await context.close()
         else:
             # Standard mode: use PagePool with persistent context
             page_info = await self._get_page(timeout, extra_headers, disable_resources, blocked_domains)
             try:
                 yield page_info
             finally:
-                await page_info.page.close()
-                self.page_pool.pages.remove(page_info)
+                if page_info.state == "error" or page_info.page.is_closed():
+                    with suppress(Exception):
+                        await page_info.page.close()
+                    self.page_pool.remove_page(page_info)
+                else:
+                    page_info.mark_ready()
 
 
 class BaseSessionMixin: