Scrapling memory usage #112

ArmandoSchiano · 2025-11-20T10:01:47Z

ArmandoSchiano
Nov 20, 2025

Hello, I have created some endpoints that call Scrapling classes. However it seems the memory usage is very high (lots of processes spawning and high RAM usage). I would like to know what's wrong with my configuration: Is there a way to close browser and context after every call?

I am using a systemd ubuntu service running simple_http_server. Below are the systemd stats 8 minutes after restart and a few calls:

user@my-server:~$ sudo systemctl status scrapling.service
● scrapling.service - Scrapling API Server
     Loaded: loaded (/etc/systemd/system/scrapling.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2025-11-20 09:09:55 UTC; 8min ago
   Main PID: 3459251 (python3)
      Tasks: 569 (limit: 28696)
     Memory: 1.4G
        CPU: 30min 42.614s
     CGroup: /system.slice/scrapling.service
             ├─3459251 /home/user/python/scrapling/venv/bin/python3 /home/user/python/scrapling/scraper_server.py
             ├─3547209 /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/node /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/package/cli.j>
             ├─3547573 /home/user/.cache/camoufox/camoufox-bin -no-remote -headless -profile /tmp/playwright_firefoxdev_profile-aCInih -juggler-pipe about:blank
             ├─3547656 /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/node /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/package/cli.j>
             ├─3547762 /home/user/.cache/camoufox/camoufox-bin -no-remote -headless -profile /tmp/playwright_firefoxdev_profile-uEMJk7 -juggler-pipe about:blank
             ├─3547780 /home/user/.cache/camoufox/camoufox-bin -contentproc -parentBuildID 20250315105650 -prefsHandle 0 -prefsLen 24826 -prefMapHandle 1 -prefMapSize 265748 -sandboxReporter 2 -chrootCli>
             ├─3547791 /home/user/.cache/camoufox/camoufox-bin -contentproc -isForBrowser -prefsHandle 0 -prefsLen 25116 -prefMapHandle 1 -prefMapSize 265748 -jsInitHandle 2 -jsInitLen 253512 -parentBuil>
             ├─3547797 /home/user/.cache/camoufox/camoufox-bin -contentproc -parentBuildID 20250315105650 -prefsHandle 0 -prefsLen 25161 -prefMapHandle 1 -prefMapSize 265748 -sandboxReporter 2 -chrootCli>
             ├─3547840 /home/user/.cache/camoufox/camoufox-bin -contentproc -isForBrowser -prefsHandle 0 -prefsLen 24376 -prefMapHandle 1 -prefMapSize 265748 -jsInitHandle 2 -jsInitLen 253512 -parentBuil>
             ├─3547975 /home/user/.cache/camoufox/camoufox-bin -contentproc -parentBuildID 20250315105650 -sandboxingKind 0 -prefsHandle 0 -prefsLen 33436 -prefMapHandle 1 -prefMapSize 265748 -sandboxRep>
             ├─3547991 /home/user/.cache/camoufox/camoufox-bin -contentproc -isForBrowser -prefsHandle 0 -prefsLen 31894 -prefMapHandle 1 -prefMapSize 265748 -jsInitHandle 2 -jsInitLen 253512 -parentBuil>
             ├─3548040 /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/node /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/package/cli.j>
             ├─3548129 /home/user/.cache/camoufox/camoufox-bin -no-remote -headless -profile /tmp/playwright_firefoxdev_profile-9FmJPb -juggler-pipe about:blank
             ├─3548418 /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/node /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/package/cli.j>
             ├─3548420 /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/node /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/package/cli.j>
             ├─3548727 /home/user/.cache/camoufox/camoufox-bin -no-remote -headless -profile /tmp/playwright_firefoxdev_profile-U573S5 -juggler-pipe about:blank
             ├─3548728 /home/user/.cache/camoufox/camoufox-bin -no-remote -headless -profile /tmp/playwright_firefoxdev_profile-qucJV6 -juggler-pipe about:blank
             ├─3548869 /home/user/.cache/camoufox/camoufox-bin -contentproc -parentBuildID 20250315105650 -prefsHandle 0 -prefsLen 24826 -prefMapHandle 1 -prefMapSize 265748 -sandboxReporter 2 -chrootCli>
             ├─3549196 /home/user/.cache/camoufox/camoufox-bin -contentproc -isForBrowser -prefsHandle 0 -prefsLen 25116 -prefMapHandle 1 -prefMapSize 265748 -jsInitHandle 2 -jsInitLen 253512 -parentBuil>
             ├─3549202 /home/user/.cache/camoufox/camoufox-bin -contentproc -parentBuildID 20250315105650 -prefsHandle 0 -prefsLen 25161 -prefMapHandle 1 -prefMapSize 265748 -sandboxReporter 2 -chrootCli>
             ├─3549251 /home/user/.cache/camoufox/camoufox-bin -contentproc -isForBrowser -prefsHandle 0 -prefsLen 24376 -prefMapHandle 1 -prefMapSize 265748 -jsInitHandle 2 -jsInitLen 253512 -parentBuil>
             ├─3549334 /home/user/.cache/camoufox/camoufox-bin -contentproc -parentBuildID 20250315105650 -sandboxingKind 0 -prefsHandle 0 -prefsLen 33436 -prefMapHandle 1 -prefMapSize 265748 -sandboxRep>
             ├─3549351 /home/user/.cache/camoufox/camoufox-bin -contentproc -isForBrowser -prefsHandle 0 -prefsLen 31894 -prefMapHandle 1 -prefMapSize 265748 -jsInitHandle 2 -jsInitLen 253512 -parentBuil>
             ├─3549932 /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/node /home/user/python/scrapling/venv/lib/python3.10/site-packages/playwright/driver/package/cli.j>
             └─3549953 /home/user/.cache/camoufox/camoufox-bin -no-remote -headless -profile /tmp/playwright_firefoxdev_profile-SKgK8S -juggler-pipe about:blank

Nov 20 09:18:31 my-server python3[3459251]: [2025-11-20 09:18:31] INFO: Fetched (200) <GET> 
Nov 20 09:18:32 my-server python3[3459251]: [2025-11-20 09:18:32]-[ReqThread_6]-[simple_http_server.http_protocol_handler:382] INFO: "POST /scrape HTTP/1.1" 200 -
Nov 20 09:18:32 my-server python3[3459251]: [2025-11-20 09:18:32]-[ReqThread_4]-[simple_http_server.http_protocol_handler:169] INFO: request version: HTTP/1.1
Nov 20 09:18:34 my-server python3[3459251]: [2025-11-20 09:18:34]-[ReqThread_5]-[simple_http_server.http_protocol_handler:169] INFO: request version: HTTP/1.1
Nov 20 09:18:34 my-server python3[3459251]: [2025-11-20 09:18:34]-[ReqThread_3]-[simple_http_server.http_protocol_handler:169] INFO: request version: HTTP/1.1
Nov 20 09:18:37 my-server python3[3459251]: [2025-11-20 09:18:37] INFO: Fetched (200) <GET> 
Nov 20 09:18:39 my-server python3[3459251]: [2025-11-20 09:18:39]-[ReqThread_7]-[simple_http_server.http_protocol_handler:382] INFO: "POST /scrape HTTP/1.1" 200 -
Nov 20 09:18:40 my-server python3[3459251]: [2025-11-20 09:18:40]-[ReqThread_8]-[simple_http_server.http_protocol_handler:169] INFO: request version: HTTP/1.1
Nov 20 09:18:41 my-server python3[3459251]: [2025-11-20 09:18:41] INFO: Fetched (404) <GET>
Nov 20 09:18:42 my-server python3[3459251]: [2025-11-20 09:18:42]-[ReqThread_1]-[simple_http_server.http_protocol_handler:382] INFO: "POST /scrape HTTP/1.1" 200 -

Below is the code:

import json
import traceback
from datetime import datetime
from typing import Dict

from scrapling.fetchers import StealthyFetcher
from simple_http_server import request_map, server, JSONBody

sessions: Dict[str, 'BrowserSession'] = {}

class BrowserSession:
    """Manages a persistent browser session with Playwright page."""

    def __init__(self, session_id: str, page, browser, target_url: str):
        self.session_id = session_id
        self.page = page
        self.browser = browser
        self.target_url = target_url
        self.created_at = datetime.now()
        self.last_activity = datetime.now()


@request_map("/scrape", method=["POST"])
async def scrape(body: JSONBody):
    """
    Scrape a webpage using Scrapling's StealthyFetcher.

    Parameters:
        target_url (str): The URL to scrape.
        config (dict): Optional configuration.
            headless (bool): Run browser in headless mode (default: True)
            adaptive (bool): Enable adaptive scraping (default: True)
            selector (str): CSS selector to extract specific elements
            selector_adaptive (bool): Enable adaptive selection (default: True)
            :param body:
    """

    target_url = body.get("target_url")

    try:
        if not body or not isinstance(body, dict):
            return json.dumps({
                "code": 400,
                "message": "Invalid Request",
                "explain": "Request body must be a valid JSON object."
            })

        if not target_url:
            return json.dumps({
                "code": 400,
                "message": "Missing Parameter",
                "explain": "Parameter[target_url] is required!"
            })
        config = body.get("config") or {}

        headless = config.get('headless', True)
        adaptive = config.get('adaptive', True)
        timeout = config.get('timeout', 30000)
        custom_config = config.get('custom_config', {})

        page = await StealthyFetcher.async_fetch(
            target_url,
            headless=headless,
            timeout=timeout,
            custom_config=custom_config,
        )

        # If a selector is provided, extract specific elements
        if 'selector' in config and config['selector']:
            selector_adaptive = config.get('selector_adaptive', True)
            elements = page.css(config['selector'], adaptive=selector_adaptive)

            # Extract text or HTML from elements
            if elements:
                results = []
                for element in elements:
                    results.append({
                        'text': element.text,
                        'html': str(element)
                    })

                return json.dumps({
                    'status': 'success',
                    'message': 'Elements extracted successfully',
                    'url': target_url,
                    'results': results
                })
            else:
                return json.dumps({
                    'status': 'success',
                    'message': 'No elements found',
                    'url': target_url,
                    'results': []
                })

        # Return full page HTML
        html = getattr(page, 'html', None) or getattr(page, 'body', None) or getattr(page, 'content', None) or str(page)

        return json.dumps({
            'status': 'success',
            'message': 'Page scraped successfully',
            'url': target_url,
            'html': html,
            "http_status_code": getattr(page, 'status_code', 200)
        })

    except Exception as e:
        return json.dumps({
            'status': 'error',
            'message': str(e),
            'traceback': traceback.format_exc(),
            'url': target_url if 'target_url' in locals() else 'unknown',
            "http_status_code": 500
        })

server.start(port=9091)

Answered by D4Vinci

Nov 22, 2025

Hi there,

The StealthyFetcher class uses more resources than DynamicFetcher by default because it uses the camoufox browser, which causes this.

Here are some things you can try:

Try to test if the stealth mode in DynamicFetcher is sufficient for your case or not. If it is, use it, as it uses much less resources.
Try to use StealthySession instead to keep the browser open, instead of opening a new browser with every request.
There's also DynamicSession for the other way around.

Let me know how these options work with you.

View full answer

D4Vinci · 2025-11-22T19:49:37Z

D4Vinci
Nov 22, 2025
Maintainer

Hi there,

The StealthyFetcher class uses more resources than DynamicFetcher by default because it uses the camoufox browser, which causes this.

Here are some things you can try:

Try to test if the stealth mode in DynamicFetcher is sufficient for your case or not. If it is, use it, as it uses much less resources.
Try to use StealthySession instead to keep the browser open, instead of opening a new browser with every request.
There's also DynamicSession for the other way around.

Let me know how these options work with you.

0 replies

Navigation Menu

Uh oh!

Scrapling memory usage #112

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Scrapling memory usage #112

Uh oh!

Uh oh!

ArmandoSchiano Nov 20, 2025

Replies: 1 comment

Uh oh!

D4Vinci Nov 22, 2025 Maintainer

ArmandoSchiano
Nov 20, 2025

D4Vinci
Nov 22, 2025
Maintainer