Skip to content

seeker

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

seeker

README.md

Seeker

Document processing pipeline for security research extraction. Fetches URLs, classifies documents via Claude, and extracts structured security information (currently: HTTP request smuggling vectors).

Quick Reference

# Install
pip install -e ".[dev]"

# Process URLs (writes manifest for summary command)
seeker process --input urls.txt --db seeker.db --manifest run.manifest.json --verbose

# Try it (needs ANTHROPIC_API_KEY + network):
seeker process --input fixtures/urls.txt --db seeker.db --verbose

# Generate summary from manifest (after running flamer)
seeker summary --manifest run.manifest.json --flamer-db flamer.db --min-id $MAX_ID

# Query extracted sections
seeker query --db seeker.db --type http_desync_vector
seeker query --db seeker.db --classification security_research

# Trace a flamer request back to its seeker source
python3 -m seeker.cli trace <flamer-request-id> --db seeker.db --flamer-db ../flamer/production.db

# Run tests
python3 -m pytest tests/ -v

Architecture

URL → Fetcher → Classifier → Chunker → Extractor → Storage
        ↓           ↓           ↓           ↓          ↓
    HTML/PDF    Claude      80K→20K     Claude     SQLite
    text        tool use    overlap     JSON

Pipeline flow:

  1. Fetch URL content (HTML, PDF, or plain text)
  2. Classify document using Claude (80K char chunk, tool use)
  3. Split text into 20K char chunks with 2K overlap
  4. Extract sections via Claude (JSON with response prefilling)
  5. Store in SQLite with content hash deduplication

Project Structure

seeker/
├── cli.py          # Entry point: process, query commands
├── config.py       # Classifications, extractors, chunk sizes
├── fetcher.py      # URL fetching (HTML/PDF/text extraction)
├── chunker.py      # Text splitting with overlap
├── classifier.py   # Document classification (Claude tool use)
├── extractor.py    # Section extraction (Claude JSON output)
└── storage.py      # SQLite operations
prompts/
├── classification.md
└── extractors/
    ├── http_desync_vector.md
    ├── default.md
    └── http_parser_code.md
tests/              # pytest tests (use :memory: SQLite)

Key Configuration (config.py)

  • CHUNK_SIZE_CLASSIFICATION = 80000 - Large input, small output
  • CHUNK_SIZE_EXTRACTION = 20000 - Smaller input, output scales with content
  • Classifications: security_research (http_desync_vector), source_code (http_parser_code), other (default)
  • Extractors: http_desync_vector for request smuggling techniques, http_parser_code for source code HTTP parsing, default for HTTP features/semantics

Development

Tech stack: Python 3.11+, anthropic SDK, httpx, BeautifulSoup4, pypdf, SQLite

Testing:

  • pytest with pytest-httpx for HTTP mocking
  • Use :memory: SQLite for test isolation
  • Mock Anthropic client with unittest.mock

Conventions:

  • Type hints: PEP 604 syntax (str | None, list[dict])
  • Docstrings: Google style
  • Commits: Conventional Commits (feat:, fix:, test:)
  • Private methods: underscore prefix (_parse_response)

Error handling: Graceful fallbacks - classifier returns "other" on error, extractor returns empty list

Database Schema

documents(id, url UNIQUE, classification, content_cache_path, fetched_at)
sections(id, document_id FK, chunk_index, section_type, title, content, content_hash, raw_llm_response)

Indexes on: section_type, content_hash, classification

Provenance Tracing

The trace command traces a flamer-generated request back to its seeker source:

python3 -m seeker.cli trace 1002368 --db seeker.db --flamer-db ../flamer/production.db

Output includes:

  • Flamer request metadata (ID, body type, timestamp)
  • Source seeker section (ID, title, URL, type)
  • Full section content
  • All prompts used with hashes (classification, extraction, flamer generation)

Finding request IDs to trace:

-- Get latest flamer request with seeker inspiration
sqlite3 ../flamer/production.db "SELECT id, inspiration_file_path FROM model_outputs_v5 WHERE inspiration_file_path LIKE 'seeker:%' ORDER BY id DESC LIMIT 5"

Pipeline Runs

Pipeline runs (./pipeline.sh) process every URL in the input file end-to-end (fetch, classify, extract) and can take a while for large URL lists, since each document requires at least two Claude calls.

Related Projects (this monorepo)

Part of the http-terminator HTTP request smuggling research pipeline:

  • seeker (this dir) — extract techniques from security-research documents
  • flamer (../flamer/) — generate test cases from extracted techniques
  • validator (../validator/) — validate generated requests in Burp Suite
  • investigator (../investigator/) — exploit/confirm/report on findings