architecture.md

Architecture

MMMeta uses a modular service architecture with a local-first control plane:

• cli and api share the same application context and service layer.
• db provides async SQLAlchemy persistence with SQLite defaulting to durable local storage.
• pipelines define reusable DAG-like stage sequences for normalization and enrichment.
• workers execute persistent queue jobs with retries, cancellation flags, and progress events.
• providers abstract OpenAI-compatible backends while preserving an offline heuristic provider for local operation.
• storage keeps artifacts in a content-addressable directory layout.
• vectorstores provide semantic search integration.
• plugins extend stages, pipelines, providers, and exporters without touching core code.

Processing flow:

1. Ingest file paths recursively and compute content hashes.
2. Normalize existing metadata and technical attributes into the unified schema.
3. Generate semantic chunks and derivative summaries.
4. Persist embeddings and search indexes.
5. Export structured archives and stream job progress to API clients.