site-auditor

A concurrent site crawler and link-health auditor

asyncio for network I/O · a process pool for HTML parsing · every performance claim backed by a benchmark in this repository

Overview · Status · Architecture · Usage · Performance · Decisions · Build log

---

Warning

Not usable yet — this is Phase 03 of 08. The repository is public from the first commit so the design and the build sequence are visible, not because there is anything to install. Nothing in this document describes working software except where the Status table says so.

No performance figure will appear here that benchmarks/ cannot reproduce.

Table of contents

• Overview
  • What it does
  • Non-goals
• Status
  • Build phases
• Architecture
  • Crawl pipeline
  • Lifecycle of a single page
  • The dependency rule
  • Module layout
• Usage
• Performance
• Engineering standards
• Design decisions
• Why this exists
• Build log
• Limitations
• License

Overview

Point it at a host. It crawls breadth-first within that host, checks the health of every link it finds, extracts and validates page metadata, and writes a report a CI pipeline can act on.

The interesting part is not the crawling. It is doing it concurrently without lying about it — bounded in-flight requests, correct cancellation, streamed output that keeps memory flat, and a benchmark honest enough to report the parts that lost.

What it does

	Capability
🕸️	Concurrent breadth-first crawl of a single host, bounded by depth and page budget
🔗	Per-link status code, final URL after redirects, and response time
🚨	Detects broken links (4xx/5xx), redirect chains, and mixed-content links
🏷️	Page metadata checks — <title>, meta description, canonical, h1 count
🤖	robots.txt compliance and per-host rate limiting
🔁	Retries transient failures with backoff and jitter; never retries a 4xx
📄	Streams JSONL to disk — memory stays flat across thousands of pages
⏹️	Ctrl-C exits cleanly, flushes partial results, returns 130
✅	Exit code 1 when links are broken, so it can gate a build

Non-goals

Deliberately out of scope — these are decisions, not a backlog

Not building	Why
Web dashboard or HTTP API	Dilutes the focus. This is a CLI tool that fits in a pipeline.
Distributed workers (Celery, Redis)	Unjustifiable complexity at this scale, and indefensible in review.
Database persistence	JSONL is the correct output for a stateless auditor.
Headless-browser JS rendering	Changes what the tool fundamentally is.
Authenticated / login-walled crawling	Out of scope for a public link auditor.
SEO scoring or sitemap generation	A different product wearing the same hat.
Entry-point plugin system	The Check protocol already covers extension.
Output formats beyond JSONL and console	Two formats, both done well.

[!NOTE] This table is the scope contract. It was written before the first commit and is not expected to grow — if something moves from here into the feature list, that is a decision worth recording in the decision log, not a quiet edit.

---

Status

Phase ▰▰▰▱▱▱▱▱ 3 / 8 complete

What works today: nothing. The repository is scaffolding and design.

#	Phase	Delivers	Status
00	Scaffold	packaging, mypy --strict, ruff, green CI	✅
01	Core model	models, protocols, URL normalisation, frontier	✅
02	Network layer	async fetcher, retry, rate limit, robots	✅
03	Crawl engine	queue, worker pool, cancellation	🚧
04	Parsing	HTML off the event loop, pluggable checks	⬜
05	Interface	streaming JSONL, console summary, CLI	⬜
06	Benchmark	four execution models over a fixed corpus	⬜
07	Harden	test coverage, docs, v1.0.0	⬜

Build phases

Phase 00 — Scaffold

A repository that does nothing, perfectly.

• uv init, src/ layout, console-script entry point
• Toolchain pinned: ruff, mypy, pytest, pytest-asyncio, httpx
• mypy --strict configured; ruff rules chosen deliberately, not copied
• errors.py — AuditError hierarchy, written before anything can raise
• GitHub Actions running lint, type-check and tests on push

Done when uv run siteaudit --version prints a version and CI is green on an empty test suite.

Phase 01 — Core model

Everything pure and synchronous, tested to the floor. No network.

• models.py — frozen dataclasses with slots=True
• protocols.py — Fetcher and Check
• urls.py — normalisation, same-host predicate, relative resolution
• frontier.py — dedup set, depth tracking, budget guard
• Adversarial URL tests: fragments, uppercase hosts, protocol-relative, mailto:, javascript:, default ports, percent-encoding

Done when test_urls.py covers 15+ hostile cases and the frontier refuses duplicates, off-host URLs and over-budget URLs.

Phase 02 — Network layer

The network, behind the Protocol.

• fetcher.py — one AsyncClient, explicit httpx.Limits, async context manager
• decorators.py — @retry with exponential backoff and jitter; policy as data
• ratelimit.py — per-host even spacing
• robots.py — fetched once, parsed, honoured

Done when FakeFetcher satisfies the Fetcher protocol without inheriting from it, and the retry test uses a fetcher that fails twice then succeeds — no mocking library anywhere.

Phase 03 — Crawl engine

The heart of the project. Budget the most time here and expect one rewrite.

• asyncio.Queue frontier, N workers inside a TaskGroup
• Semaphore in-flight bound; asyncio.timeout() per request
• Correct termination: await queue.join() then cancel; task_done() in finally
• Graceful cancellation on KeyboardInterrupt, partial results flushed
• Developed under debug=True, every slow-callback warning treated as a defect

Done when a 50-page crawl runs entirely off FakeFetcher, Ctrl-C exits clean with partial output and code 130, and a cancellation test passes in CI.

Phase 04 — Parsing

The CPU half, where the GIL stops being theory.

• parsing.py — pure, picklable, dependency-free functions
• ProcessPoolExecutor via run_in_executor, size configurable
• checks/links.py and checks/meta.py, both structurally matching Check
• --no-process-pool flag so both paths stay benchmarkable

Done when adding a third check requires editing cli.py only, and both execution paths produce byte-identical reports.

Phase 05 — Interface

Usable by someone who is not me, including a machine.

• Async JSONL writer consuming a results queue, one record per line
• Console summary: counts by status class, slowest pages, broken links
• cli.py as composition root — parse, construct, inject, run
• Exit codes wired through

Done when it runs from a clean clone and memory stays flat across a 500-page crawl — measured, not assumed.

Phase 06 — Benchmark

The artifact that turns a claim into evidence.

• Fixed local corpus, served over loopback
• Four execution models over the identical workload
• Wall clock and peak memory, median of five runs
• RESULTS.md — the table, then the explanation

Done when the table reproduces on a second machine and the write-up is honest about anything that lost.

Phase 07 — Harden

The phase most people skip, and the one that gets read.

• Test gaps closed — delete any module and something fails
• Architecture documented, decision log complete
• v1.0.0 tagged, clean clone verified

Done when every README claim traces to a test or a benchmark row.

---

Architecture

Crawl pipeline

flowchart LR
    A(["Seed URL"]) --> F1

    subgraph FR["Frontier"]
        direction TB
        F1["Dedup set"]
        F2["asyncio.Queue"]
        F1 --> F2
    end

    F2 --> W

    subgraph WP["Worker pool - asyncio.TaskGroup"]
        W["Worker x N<br/>Semaphore bounded<br/>per-host rate limit"]
    end

    W --> FE["Fetcher<br/>one AsyncClient for the run"]
    FE --> PP["Process pool<br/>HTML parsing off the loop"]
    PP --> CH["Checks<br/>links, metadata"]
    CH --> WR[("report.jsonl<br/>streamed")]
    CH -.->|"new in-host links"| F1

    BG{{"Budget guard<br/>depth, page cap, robots, same-host"}}
    BG -.-> F1

Loading

Lifecycle of a single page

sequenceDiagram
    autonumber
    participant F as Frontier
    participant W as Worker
    participant R as RateLimiter
    participant H as Fetcher
    participant P as ProcessPool
    participant O as Writer

    F->>W: url via await queue.get
    W->>R: acquire host token
    R-->>W: granted
    W->>H: GET inside asyncio.timeout
    H-->>W: response, final_url, elapsed
    W->>P: run_in_executor parse html
    P-->>W: links plus metadata
    W->>O: result record
    W->>F: enqueue new in-host links
    W->>F: task_done in finally

Loading

Important

task_done() lives in a finally block. One exception on the path to it and await queue.join() never returns — the crawler hangs with no output and no traceback. This is the single most likely bug in the whole project.

The dependency rule

Why crawler.py imports nothing concrete

crawler.py imports protocols.py and nothing else from this package — not the fetcher, not the checks, not the report writer. cli.py is the only module permitted to construct real implementations and inject them.

        protocols.py
       ↗      ↑      ↖
crawler.py  fetcher.py  checks/
       ↖      ↑      ↗
           cli.py            ← the only place concretes are wired

Two things this buys, both testable:

1. The engine runs with no network. A FakeFetcher returning canned HTML from a dict satisfies the protocol structurally — no inheritance, no mocking library.
2. Implementations swap without an edit. The engine cannot know or care.

If crawler.py grows a concrete import, that is a design regression, not a convenience.

Module layout

Repository structure

site-auditor/
├── src/siteaudit/
│   ├── cli.py          composition root — the only place concretes are wired
│   ├── models.py       Page, Link, CheckResult, AuditReport
│   ├── protocols.py    Fetcher, Check — the abstraction seam
│   ├── errors.py       AuditError hierarchy
│   ├── urls.py         normalisation, same-host predicate (pure, sync)
│   ├── frontier.py     dedup set, depth tracking, budget guard
│   ├── robots.py       robots.txt fetch and verdict
│   ├── ratelimit.py    per-host token bucket
│   ├── decorators.py   @retry with backoff and jitter, @timed
│   ├── fetcher.py      httpx.AsyncClient wrapper, async context manager
│   ├── crawler.py      the engine — queue, workers, TaskGroup
│   ├── parsing.py      pure CPU functions, picklable, no package imports
│   ├── checks/
│   │   ├── links.py    broken links, redirect chains, mixed content
│   │   └── meta.py     title, description, canonical, h1 count
│   └── report.py       async JSONL writer, console summary
├── benchmarks/
│   ├── corpus/         fixed local pages, served over loopback
│   ├── bench.py        four execution models, one workload
│   └── RESULTS.md      the table and the explanation
└── tests/
    ├── fakes.py        FakeFetcher — no mocking library
    ├── test_urls.py    the hardest-tested module in the repo
    ├── test_crawler.py including a cancellation test
    └── ...

---

Usage

Note

Not functional yet. Recorded here so the interface is settled before it is built.

uv sync

uv run siteaudit https://example.com \
    --max-depth 3 \
    --max-pages 500 \
    --concurrency 20 \
    --out report.jsonl

All flags

Flag	Default	Purpose
--max-depth	3	How far from the seed URL to crawl
--max-pages	500	Hard ceiling on pages admitted to the crawl. Failed fetches still count against it
--concurrency	20	In-flight request bound
--timeout	10	Per-request deadline, seconds
--rate	5	Requests per second, per host
--out	report.jsonl	Streamed output path
--no-process-pool	off	Parse inline — for benchmark comparison
--user-agent	siteaudit/1.0	Sent on every request

Output format

One JSON object per line, written as results arrive — never accumulated in memory.

{"url":"https://example.com/pricing","status":200,"final_url":"https://example.com/pricing","elapsed_ms":84.2,"depth":1,"checks":[]}
{"url":"https://example.com/old","status":301,"final_url":"https://example.com/new","elapsed_ms":61.7,"depth":2,"checks":[{"check":"links","severity":"warn","message":"redirect chain length 2"}]}
{"url":"https://example.com/gone","status":404,"final_url":"https://example.com/gone","elapsed_ms":39.1,"depth":2,"checks":[{"check":"links","severity":"error","message":"broken link"}]}

Exit codes

Code	Meaning
0	Crawl completed, no broken links
1	Crawl completed, broken links found
2	Usage error
130	Interrupted — partial results written

Which makes it usable as a build gate:

- name: Audit links
  run: uv run siteaudit https://staging.example.com --max-pages 300

---

Performance

Note

Awaiting Phase 06. The protocol below is fixed in advance so the results cannot be shaped after the fact.

Model	Implementation	Wall clock	Peak memory
A · Sequential	one request at a time	—	—
B · Threaded	ThreadPoolExecutor	—	—
C · Async	AsyncClient, bounded	—	—
D · Async + processes	C, parsing in a process pool	—	—

Measurement protocol

• Local corpus of static pages served over loopback — never a live third-party site, which is neither reproducible nor polite
• Median of five runs, min and max reported
• Wall clock and peak memory, alongside machine, Python version and corpus size
• Request count identical across all four models, or the comparison is void
• Clock stopped before any output is printed

Tip

Whether the process pool is worth its pickling overhead is an open question this benchmark exists to answer — including if the answer is no. A measured loss with a stated crossover point is a better result than an unverified win.

---

Engineering standards

Held from Phase 00, not retrofitted at the end.

Gate	Tool	Standard
Types	mypy --strict	Zero errors. Every # type: ignore carries a comment explaining itself.
Lint	ruff	Clean. Rules chosen deliberately, not copied.
Tests	pytest, pytest-asyncio	Delete any module and at least one test fails.
CI	GitHub Actions	Green on every push. Failing tests block the merge.
Packaging	uv, src/ layout	uv sync on a clean clone is the entire setup.

Local development

git clone https://github.com/subham-hq/site-auditor.git
cd site-auditor
uv sync

uv run pytest                 # tests
uv run mypy --strict src/     # type check
uv run ruff check .           # lint
uv run ruff format .          # format

python benchmarks/bench.py    # benchmark (Phase 06+)

---

Design decisions

An append-only log. One entry per real trade-off, written at the moment it is made — not reconstructed at the end.

ADR-000 · Template — copy this for each new entry

Context — what forced a choice.

Decision — what was chosen.

Alternatives rejected — what else was on the table, and why it lost.

Trade-off accepted — what this costs. Every decision costs something; an entry with no cost listed is not finished.

Status — accepted · superseded by ADR-00X

ADR-001 · Response is a local type, not httpx.Response

Context — crawler.py needs a response object to work with. The obvious choice is httpx.Response, since httpx is already the HTTP client.

Decision — Define a small frozen Response dataclass in models.py with six fields: url, final_url, status, elapsed_ms, content_type, text. fetcher.py adapts httpx.Response into it.

Alternatives rejected — Returning httpx.Response directly from the Fetcher protocol. Simpler and better typed, but the protocol would then name a third-party type, so crawler.py would transitively depend on httpx and the dependency rule would be weaker than the README claims. Test fakes would also have to construct real HTTP objects — headers, encodings — to exercise a queue.

Trade-off accepted — A small adapter in fetcher.py, and no access to httpx features not projected here. Adding a field is cheap; unpicking the coupling later would not be.

Status — accepted

ADR-002 · Subdomains are distinct hosts

Context — is_same_host bounds the crawl: given a seed, it decides which discovered links are in scope. It compares .hostname, so www.example.com and example.com are two different hosts. That is strictly correct — they are different names — but on most sites they serve the same content, and many sites redirect one to the other.

Decision — Keep the strict comparison, and handle the redirect case with rebind_seed. The seed is validated at construction from what the user typed; if the first fetch redirects, rebind_seed re-points the host check at the seed's final URL, so every later comparison is made against the host the site actually canonicalises to. It refuses to run once any link has been added, so a crawl cannot change hosts partway through.

Alternatives rejected — Stripping a leading www. before comparing would fix the common case in one line, but it is a guess about site structure, and wrong for any site where subdomains serve genuinely different content. Comparing registrable domains is the more general version and needs the Public Suffix List: naive last-two-labels turns bbc.co.uk into co.uk, so every British domain becomes same-host. That is a real dependency for a marginal gain.

Trade-off accepted — A seed of example.com will not follow links to www.example.com unless a redirect points it there first. On a site that canonicalises to www but is reachable at the bare domain without redirecting, the crawl finds nothing beyond the seed. The skipped_off_host counter is what makes that visible rather than silent.

Status — accepted

ADR-003 · protego over the stdlib robots parser

Context — robots.py needs to answer whether a URL is permitted. Python ships urllib.robotparser, so the zero-dependency option was tried first.

Its entire matching logic is filename.startswith(self.path) — prefix matching, nothing more. No * wildcards, no $ end-anchor, both of which are in RFC 9309 and both of which real sites rely on. Tested against GitHub's live robots.txt:

Rule	URL	stdlib	protego
Disallow: /copilot/	/copilot/	blocked	blocked
Disallow: /search$	/search	allowed	blocked
Disallow: /*q=	/search?q=x	allowed	blocked
Disallow: /*/*/pulse	/a/b/pulse	allowed	blocked

GitHub's file is almost entirely wildcard rules, so the stdlib would have honoured a handful and ignored the rest — while this README claimed robots.txt compliance.

Decision — Use protego, which implements RFC 9309: wildcards, the $ anchor, and longest-match precedence.

Alternatives rejected — Keeping the stdlib. Not a trade of correctness for convenience but a silent failure: nothing raises, the crawl looks clean, and pages the site owner disallowed get fetched anyway. Hand-rolling the matching was never considered — the precedence rules are subtle enough that getting them wrong would reproduce exactly the problem being fixed.

Trade-off accepted — A third-party dependency where the stdlib would run. Mitigated by protego shipping py.typed, so mypy --strict passes without stubs, and by the parser being confined to one module: swapping it again would touch robots.py and nothing else.

Status — accepted

---

Why this exists

🔒 To be written.

---

Build log

Newest first. One entry per phase — what shipped, and what it cost.

Date	Phase	Shipped	Notes
2026-08-19	02	Fetcher, retry, rate limit, robots	Stdlib robots parser silently allowed every wildcard rule; swapped to protego
2026-08-03	01	Models, protocols, urls, frontier	is_same_host let mailto: through — None == None
2026-08-03	00	Scaffold, toolchain, CI	pyproject missing [build-system]; package never installed

Entry template

| 2026-08-14 | 03 | Crawl engine — queue, workers, cancellation | Rewrote termination twice; queue.join() vs awaiting workers |

Keep the notes column honest. "Rewrote it twice" is more interesting than "done".

---

Limitations

🔒 To be written after Phase 07.

---

License

MIT — see LICENSE.

_{Built as the concurrency and performance pillar of a deliberate backend engineering track.}