|
| 1 | +# Building RAG Systems |
| 2 | + |
| 3 | +RAG pipelines are only as good as the text you feed them. Raw HTML wastes tokens on markup, navigation, scripts, and tracking noise, and it can even carry hidden prompt-injection content straight into your LLM. Scrapling turns pages and whole websites into clean, sanitized Markdown with no LLM in the loop, so your ingestion runs fast and costs nothing per page. |
| 4 | + |
| 5 | +## Installation |
| 6 | + |
| 7 | +```bash |
| 8 | +pip install "scrapling[rag]" |
| 9 | + |
| 10 | +scrapling install |
| 11 | +``` |
| 12 | + |
| 13 | +The `rag` extra installs the fetchers with Markdown conversion support (the `ai`, `shell`, and `all` extras include it too). The `scrapling install` command downloads the browser dependencies, which you only need for the browser-based fetchers. |
| 14 | + |
| 15 | +## One page to Markdown |
| 16 | + |
| 17 | +Every [Response](fetching/choosing.md) has a `markdown()` method: |
| 18 | + |
| 19 | +```python |
| 20 | +from scrapling.fetchers import Fetcher |
| 21 | + |
| 22 | +markdown = Fetcher.get("https://example.com").markdown(main_content_only=True) |
| 23 | +``` |
| 24 | + |
| 25 | +It works with all fetchers, so pages behind Cloudflare are one line away too: |
| 26 | + |
| 27 | +```python |
| 28 | +from scrapling.fetchers import StealthyFetcher |
| 29 | + |
| 30 | +markdown = StealthyFetcher.fetch("https://protected.example.com", solve_cloudflare=True).markdown(main_content_only=True) |
| 31 | +``` |
| 32 | + |
| 33 | +Two arguments control the output: |
| 34 | + |
| 35 | +- `main_content_only`: Convert only the content inside the page's `<body>` tag. |
| 36 | +- `css_selector`: Convert only the elements matching a CSS selector (all matches are concatenated). Use it to extract exactly the part your pipeline needs and save tokens: |
| 37 | + |
| 38 | +```python |
| 39 | +markdown = Fetcher.get("https://example.com/docs/page").markdown(css_selector="article") |
| 40 | +``` |
| 41 | + |
| 42 | +Whatever you pass, scripts, styles, and hidden content are always removed before conversion. This is the same cleaning the [MCP server](mcp-server.md) uses to protect AI agents from prompt injection: CSS-hidden elements, `aria-hidden` elements, `<template>` tags, HTML comments, and zero-width characters never reach your model. |
| 43 | + |
| 44 | +## A whole website to a Markdown corpus |
| 45 | + |
| 46 | +The `SiteToMarkdownSpider` template crawls a website and converts every page, powered by the [spiders framework](spiders/architecture.md), so you get concurrency, autothrottle, robots.txt compliance, and pause/resume for free: |
| 47 | + |
| 48 | +```python |
| 49 | +from scrapling.spiders import SiteToMarkdownSpider |
| 50 | + |
| 51 | +class DocsSpider(SiteToMarkdownSpider): |
| 52 | + name = "docs" |
| 53 | + start_urls = ["https://example.com/docs/"] |
| 54 | + allowed_domains = {"example.com"} |
| 55 | + output_dir = "docs_markdown" |
| 56 | + max_pages = 200 |
| 57 | + |
| 58 | +result = DocsSpider().start() |
| 59 | +result.items.to_jsonl("docs.jsonl") |
| 60 | +``` |
| 61 | + |
| 62 | +Each crawled page becomes one item with `url`, `title`, and `markdown` keys. With `output_dir` set, each page is also written to a Markdown file named after its URL, so the run above gives you both a folder of `.md` files and a `docs.jsonl` ready for ingestion. |
| 63 | + |
| 64 | +The template requires `allowed_domains` so the crawl stays bound to the target website. The options: |
| 65 | + |
| 66 | +- `css_selector` / `main_content_only`: Passed to `markdown()` for every page, with `main_content_only` enabled by default. |
| 67 | +- `output_dir`: When set, writes one Markdown file per page. |
| 68 | +- `max_pages`: Maximum number of pages to convert. Requests already queued when the cap hits may still be fetched, but they aren't converted. `0` (the default) disables it. |
| 69 | + |
| 70 | +Every page link inside `allowed_domains` is followed by default. Since the template builds on [CrawlSpider](spiders/generic-templates.md), override `rules()` with your own [LinkExtractor](spiders/generic-templates.md) to control the crawl: `allow` narrows it to the URL patterns you want, and `deny` drops the patterns you don't (login pages, tag listings, print views, etc.): |
| 71 | + |
| 72 | +```python |
| 73 | +from scrapling.spiders import CrawlRule, LinkExtractor, SiteToMarkdownSpider |
| 74 | + |
| 75 | +class DocsSpider(SiteToMarkdownSpider): |
| 76 | + name = "docs" |
| 77 | + start_urls = ["https://example.com/"] |
| 78 | + allowed_domains = {"example.com"} |
| 79 | + |
| 80 | + def rules(self): |
| 81 | + return [CrawlRule(LinkExtractor(allow=r"/docs/", deny=[r"/docs/changelog/", r"\?print="]))] |
| 82 | +``` |
| 83 | + |
| 84 | +`deny` wins over `allow`, and a rule can carry a `priority` or a `process_request` hook as with any **CrawlSpider**. |
| 85 | + |
| 86 | +## Feeding a vector store |
| 87 | + |
| 88 | +The JSONL output plugs into any embedding pipeline. A minimal example: |
| 89 | + |
| 90 | +```python |
| 91 | +import json |
| 92 | + |
| 93 | +with open("docs.jsonl") as f: |
| 94 | + for line in f: |
| 95 | + page = json.loads(line) |
| 96 | + for chunk in split_into_chunks(page["markdown"]): |
| 97 | + vector_store.add(text=chunk, metadata={"url": page["url"], "title": page["title"]}) |
| 98 | +``` |
| 99 | + |
| 100 | +Use `css_selector` on the spider to cut boilerplate before chunking instead of cleaning it downstream. The less noise you embed, the better your retrieval. |
| 101 | + |
| 102 | +## Interactive alternatives |
| 103 | + |
| 104 | +For conversational scraping instead of pipelines, the [MCP server](mcp-server.md) gives your AI chatbot the same Markdown extraction as tools, and this agent skill teaches coding agents to write this code themselves. |
0 commit comments