deepdig

deepdig is a local-first research frontier CLI. It is designed to crawl a bounded source frontier, preserve evidence in SQLite, and produce an exportable bundle for later use with a local coding model.

Current status: early implementation. The CLI supports seed URL crawling, local document ingestion, sitemap discovery, arXiv, GitHub issue search, OpenAlex, Stack Exchange, SearXNG-backed mailing-list discovery, llms.txt and markdown mirror probing, SQLite frontier persistence, FTS5 plus local sparse/semantic-vector chunk search, basic novelty scoring, near-duplicate detection, LM Studio planning/report calls, structured finding-card extraction, status inspection, resume, local retrieval, and bundle export.

Local model

The default model endpoint is LM Studio compatible:

deepdig doctor --llm-base-url http://localhost:1234
deepdig doctor --llm-base-url http://192.168.1.36:1234

If no model name is supplied, deepdig tries to discover one from /v1/models.

Basic use

deepdig run --goal "find obscure implementation strategies for local AI research tools" --seed-url https://example.com --max-urls 1000 --min-sources 1000 --max-depth 2
deepdig status --run runs/<run-dir>
deepdig report --run runs/<run-dir>
deepdig export --run runs/<run-dir> --out exports/<bundle-name>

For SearXNG seed discovery:

deepdig run --goal "..." --searxng-url http://localhost:8080 --max-urls 100

For sitemap discovery:

deepdig run --goal "..." --sitemap-url https://example.com --max-urls 1000 --min-sources 1000

For local PDFs or documents:

deepdig run --goal "..." --local-document C:\path\to\paper.pdf --min-sources 1000

Document conversion uses Microsoft MarkItDown when importable. This repo expects the cloned reference at ../kb/markitdown or an installed markitdown package. Text-like local files work without extra packages.

For source connectors:

deepdig run --goal "..." --arxiv --github --openalex --stackexchange --max-urls 5000 --min-sources 3000
deepdig run --goal "..." --arxiv-query "local rag sqlite fts5" --github-query "local crawler frontier retry bug"
deepdig run --goal "..." --searxng-url http://localhost:8080 --mailing-list-query "sqlite fts5 tokenizer regression"

Connector flags without explicit queries reuse the LLM-generated plan queries. arXiv seeds PDF URLs where available so the document converter can preserve paper text.

For extra-deep runs with source diversity controls:

deepdig run --goal "..." --searxng-url http://localhost:8080 --arxiv --github --openalex --stackexchange --max-urls 10000 --min-sources 10000 --source-type-limit page-link=5500 --source-type-limit github-issue=3500 --domain-limit github.com=4500 --claim-limit 80 --claim-chunk-limit 24 --claim-time-budget 3600

By default, deepdig probes discovered HTTP(S) domains for /llms.txt, /llms-full.txt, /full-documentation.txt, and likely .md mirrors. Use --no-llm-docs to disable that. Use --playwright-fallback when you want JavaScript-heavy pages rendered with Playwright if it is installed locally.

To query the local RAG store:

deepdig retrieve --run runs/<run-dir> --query "obscure frontier retry workaround" --limit 10

Retrieval results include retrieval_score and retrieval_signals, so a local coding model can see whether evidence came from BM25, sparse vectors, semantic vectors, or fallback novelty.

The default run target is intentionally deep: --max-urls 1000 and --min-sources 1000. If --min-sources is greater than --max-urls, the effective target becomes the minimum. If the frontier is exhausted before the minimum is met, the run status becomes frontier_exhausted_before_min_sources.

For reproducible runs, use a JSON config:

{
  "max_urls": 5000,
  "min_sources": 3000,
  "max_depth": 3,
  "seed_urls": ["https://example.com"],
  "local_documents": ["C:\\path\\to\\paper.pdf"],
  "sitemap_urls": ["https://example.com"],
  "searxng_url": "http://localhost:8080",
  "arxiv_enabled": true,
  "arxiv_queries": ["local ai research agent pdf"],
  "github_enabled": true,
  "github_queries": ["local crawler frontier retry issue"],
  "openalex_enabled": true,
  "stackexchange_enabled": true,
  "mailing_list_queries": ["sqlite fts5 tokenizer bug"],
  "discover_llm_docs": true,
  "playwright_fallback": false,
  "source_type_limits": {
    "page-link": 5500,
    "github-issue": 3500
  },
  "domain_limits": {
    "github.com": 4500
  },
  "request_delay": 1.0,
  "claim_score_threshold": 8.0,
  "claim_limit": 50,
  "claim_chunk_limit": 24,
  "claim_time_budget_seconds": 1800
}

deepdig run --goal "..." --config research.json

Design

The crawler writes every durable state transition to research.sqlite, so interrupted runs can be resumed. Crawling is intentionally polite by default: robots.txt is checked for HTTP(S), requests are rate-limited per host, and retries use backoff.

Each run also writes:

• run_config.json: effective config after CLI and file config are merged
• metrics.json: source minimum progress, useful chunk rates, frontier counts, and claim counts
• report.md: final synthesis from hybrid retrieval
• events.jsonl: connector, crawl, conversion, and progress events for long-run checkpoint visibility
• raw/ and markdown/: preserved source evidence