Knowledge Extractor

A modular browser-based platform for extracting structured knowledge from web sources into a normalized, portable format. Instagram Saved Posts is the first supported source; the architecture supports additional connectors without modifying the core engine.

Motivation

Useful information is trapped inside social media platforms, videos, screenshots, and PDFs. Knowledge Extractor transforms it into structured, searchable, AI-ready data that belongs to the user. It extracts and normalizes — it does not automate engagement, circumvent access controls, or replace official APIs.

Architecture overview

The system is a layered monorepo built around a single normalized domain model (IResource). Every source is handled by an isolated connector that discovers, extracts, and normalizes content. The platform engine provides storage, scheduling, diagnostics, and export — all source-agnostic.

Content Script (DOM Adapter)
  → Navigator (browser manipulation)
  → Connector.extract (StrategyChain → Parser)
  → Connector.normalize (→ IResource)
  → CrawlController (orchestration)
  → Scheduler (queue, retry, backoff)
  → Storage (IndexedDB resources + OPFS media)
  → Enrichment (offscreen-document OCR)
  → Export (JSON / Markdown / Obsidian, incremental)
  → MetricsCollector + DiagnosticsCollector
  → Popup (stateless monitoring dashboard)

See docs/architecture/ARCHITECTURE.md for the full architecture reference with diagrams.

Repository layout

packages/
  types/        Layer 0 — interfaces, domain model, contracts
  shared/       Layer 1 — logging, metrics, diagnostics, feature flags
  storage/      Layer 2 — IndexedDB storage engine + OPFS media store
  export/       Layer 2 — pure export projection + serializers

connectors/
  instagram/    Instagram connector (implemented — Alpha)
  reddit/       Planned (placeholder)
  linkedin/     Planned (placeholder)
  youtube/      Planned (placeholder)
  x/            Planned (placeholder)
  pdf/          Planned (placeholder)
  web/          Planned (placeholder)

apps/
  extension/    Chrome MV3 extension (background, content, popup)
  desktop/      Planned (placeholder)

docs/
  architecture/ ARCHITECTURE.md, PIPELINE.md, CONNECTOR_SYSTEM.md, STORAGE.md
  guides/       CONNECTOR_GUIDE.md, CONTRIBUTING.md, DEVELOPMENT.md, TESTING.md
  verification/ Alpha report, phase walkthroughs, navigation evaluation
  rfc/          RFC-0001 (Alpha Stabilization)
  archive/      Superseded historical documents

Current status

Stage: Beta-3 complete (knowledge ownership & export)

Completed engineering work:

• Monorepo infrastructure (pnpm, Turborepo, ESLint, Prettier, dependency-cruiser, Husky)
• Domain contracts and normalized resource model
• Connector-agnostic crawl runtime: orchestration consumes the IRuntimeConnector contract via a ConnectorRegistry; adding a source is a new connector package + one registration line + a manifest match
• Instagram connector (the first IRuntimeConnector implementation) with three-strategy extraction chain, surface-aware navigation, carousel traversal, and incremental discovery
• Chrome MV3 extension with MV3-safe orchestration (pinned-tab crawl, self-scheduling loop, chrome.alarms watchdog)
• Durable persistence: IndexedDB (resources, control state) + OPFS (media bytes)
• In-crawl media capture with policy-managed retention (MediaJanitor)
• Offscreen-document OCR enrichment (Tesseract.js)
• Export pipeline: JSON (NDJSON), Markdown, and Obsidian vault targets, with incremental (watermark-based) export and media inclusion policies
• Full diagnostics and metrics wiring; live smoke harness (SMOKE.md)
• Stateless popup dashboard with crawl controls and export panel

All CI gates pass: build, typecheck, lint, test, depcruise, format.

Getting started

# Prerequisites: Node.js ≥20, pnpm 9
pnpm install
pnpm run setup:ocr    # one-time: downloads eng.traineddata (~22 MB) for offline OCR
pnpm run build

# Load the extension in Chrome
# 1. Open chrome://extensions
# 2. Enable Developer Mode
# 3. Click "Load unpacked" → select apps/extension/dist
# 4. Navigate to instagram.com/saved
# 5. Click the extension popup → Start

See docs/guides/DEVELOPMENT.md for the full development workflow.

Development

pnpm run typecheck    # TypeScript across all packages
pnpm run lint         # ESLint across all packages
pnpm run test         # Vitest (connector fixtures)
pnpm run build        # Vite build (extension)
pnpm run depcruise    # Architecture enforcement
pnpm run format:check # Prettier

Packaging

pnpm run setup:ocr          # ensure the OCR language asset is present
pnpm run build
pnpm run package:extension  # → release/knowledge-extractor-<version>.zip

The zip contains the extension with manifest.json at its root — extract and "Load unpacked", or upload it to the Chrome Web Store. Packaging fails if the OCR asset is missing (pass --allow-no-ocr for a crawler-only build).

Roadmap

See ROADMAP.md for the full Alpha → Beta → v1 roadmap.

Beta-3 (knowledge ownership & export) is complete. The standing validation procedure is the live smoke harness and runbook in SMOKE.md; the next phase (Beta-4) is in planning — see .claude/docs/40_NEXT_TASK.md.

Current limitations

• Only Instagram is implemented (the first connector); other connectors are placeholders.
• OCR requires the English language asset fetched by pnpm run setup:ocr before building; without it, resources persist but are never ENRICHED.
• Export ZIPs are STORE-only (no compression).
• The typed event/message bus is not yet enforced end-to-end (raw string actions remain).
• No semantic/keyword search over extracted content yet.