Project specifications following Spec-Driven Design adapted to hexagonal architecture.
This directory contains the structural documentation of OpenArg reverse-engineered from the code: it describes what each component does, how it is built today, and what tech debt exists. It does not modify code; it only documents reality.
Going forward it serves as the basis for forward SDD: new features are specified before being coded, reusing the conventions established here.
Contributor rule: any PR that changes observable behavior must update the relevant spec.md / plan.md as part of the change. See the README § Spec-Driven Design for the full contract.
The top-level modules are ordered roughly by architectural depth. Modules with ≥3 distinct responsibilities are decomposed into sub-modules — in those cases the top-level spec.md is a compact index and detailed FRs / tech debt live in the child folders.
| # | Module | Sub-modules | Scope |
|---|---|---|---|
| 000 | 000-architecture/ |
— | Macro as-built: layers, DI, routing, auth inventory, data API endpoints. |
| 001 | 001-query-pipeline/ |
5 sub-modules | LangGraph query pipeline — 16 nodes + 1 subgraph. |
| 001a | 001-query-pipeline/001a-intake/ |
— | Classification, cache check, memory load, query preprocessing. |
| 001b | 001-query-pipeline/001b-planning/ |
— | Planner + skill detection + coordinator + replan loop. |
| 001c | 001-query-pipeline/001c-execution/ |
— | Step executor, connector dispatch, fallback injection. |
| 001d | 001-query-pipeline/001d-analysis/ |
— | Analyst + policy synthesis, chart/map building, META parsing. |
| 001e | 001-query-pipeline/001e-finalization/ |
— | Finalize node, background tasks, stream close. |
| 002 | 002-connectors/ |
16 sub-modules | Connector pattern + every specific connector (BCRA, Series Tiempo, CKAN, DDJJ, ...). |
| 003 | 003-auth/ |
— | Auth inventory: JWT NextAuth, BACKEND_API_KEY service token, user API keys (oarg_sk_*), ADMIN_EMAILS allowlist, DATA_SERVICE_TOKEN (svc_xxx). |
| 004 | 004-semantic-cache/ |
— | pgvector semantic cache + Redis exact cache + TTL by intent + cleanup beat task. |
| 005 | 005-vector-search/ |
— | Hybrid retrieval: vector + lexical + RRF fusion. |
| 006 | 006-datasets/ |
2 sub-modules | Dataset catalog + ingestion pipeline. |
| 006a | 006-datasets/006a-catalog/ |
— | Metadata, entity, IDatasetRepository, listing/stats/download endpoints. |
| 006b | 006-datasets/006b-ingestion/ |
— | Scraper → collector → embedder → S3; 5-chunk strategy; _detect_schema_drift(). |
| 007 | 007-public-api/ |
— | /api/v1/ask Bearer-token public endpoint + per-plan rate limiting. |
| 008 | 008-developers-keys/ |
— | API key CRUD for end users (SHA-256 hash, once-visible plaintext). |
| 009 | 009-monitoring/ |
— | Health checks + in-memory MetricsCollector + /metrics. |
| 010 | 010-sandbox-sql/ |
2 sub-modules | SQL sandbox + NL2SQL. |
| 010a | 010-sandbox-sql/010a-sql-sandbox/ |
— | Read-only executor, 3-layer validation, ThreadPoolExecutor. |
| 010b | 010-sandbox-sql/010b-nl2sql/ |
— | LLM → SQL → execute → fix loop (subgraph dead code, see FIX-004). |
| 011 | 011-table-catalog/ |
— | Vector-indexed cached table metadata for NL2SQL grounding (legacy; supplanted by 015). |
| 012 | 012-admin/ |
— | Admin task registry + orchestrator endpoints. |
| 013 | 013-ingestion-validation/ |
— | WS0 — IngestionValidator + 14 detectors + 3 modes (pre/post/retro) + ingestion_findings audit trail. |
| 014 | 014-state-machine/ |
— | WS0.5 — explicit cached_datasets state machine + invariant enforcer + error_category taxonomy. |
| 015 | 015-catalog-resources/ |
— | WS2 / WS3 / WS4 — logical catalog (catalog_resources) + deterministic naming + hybrid serving. |
| 016 | 016-serving-port/ |
— | IServingPort — the medallion-aware discovery surface the pipeline reads instead of physical table names. |
| 017 | 017-raw-layer/ |
— | The raw schema: one immutable table per ingest, <portal>__<slug>__<hash>__v<N>, tracked in raw_table_versions. |
| 018 | 018-contracts-staging/ |
— | DEPRECATED 2026-05-06 — the contracts + staging layer (medallion L2). Superseded by 019; content kept for history. |
| 019 | 019-marts/ |
— | Marts: curated materialized views built from raw.* via SQL macros. The surface the chat actually serves. |
| 020 | 020-legacy-pipeline-tests-migration/ |
— | Retirement of smart_query_service and its test suite once the LangGraph pipeline became the only path. |
| 021 | 021-parser-hardening/ |
— | Parser fixes applied in-place over already-ingested tables (parse_repair), plus the PDF parser. |
| 022 | 022-mart-quality/ |
— | Nightly quality audit over every built mart + the serving_blocked switch. |
| 023 | 023-schema-snapshots/ |
— | Preserve a table's shape and value profile before any audited drop, so a format change leaves evidence. |
| 024 | 024-drift-classification/ |
— | The exoneration cascade: gates that can only prove a shape change was not upstream drift. |
| 025 | 025-self-repair/ |
— | Routing a failure to the artefact that is actually wrong: reinduction repairs one table, APR only ever proposes a parser change. |
| 026 | 026-dataset-refresh/ |
— | Reading a source more than once. A dataset is collected once and never again, which makes drift undetectable and answers three months old. |
| 027 | 027-catalog-integrity/ |
— | Making the registry's answers true, and deciding what a resource is when a portal renames it. A registry that lies does not contradict itself. |
Cross-cutting artifacts:
| Doc | Purpose |
|---|---|
constitution.md |
Non-negotiable principles (hexagonal, DI, async-first, anti-hallucination, ports before adapters). |
FIX_BACKLOG.md |
Prioritized list of fixes discovered during spec review (8 items, 5 applied, 3 pending). |
REVIEW_REPORT_2026-04-10.md |
Senior-engineer review pass cross-checking specs against code. |
Each module has two files: spec.md (the WHAT/WHY) and plan.md (the HOW). The separation follows GitHub Spec Kit and respects the hexagonal boundary: spec.md lives in the domain language, plan.md speaks of infrastructure.
When a module is large enough to warrant sub-modules, its top-level spec.md and plan.md become compact indices that list sub-modules and point to them for detail. The sub-modules each have their own spec.md + plan.md following the standard template.
# Spec: <Name>
**Type**: Reverse-engineered | Forward
**Status**: Draft | Reviewed | Canonical
**Last synced with code**: YYYY-MM-DD
**Hexagonal scope**: Domain | Domain+App | Full-stack | Adapter-only
## 1. Context & Purpose
One paragraph. What this feature does for the user/system. Domain level, no technology.
## 2. Ubiquitous Language
Terms with their definitions. Domain only.
## 3. User Stories
Prioritized stories (P1/P2/P3). Each independently testable.
- **US-NNN (Pn)**: As a <role>, I want <action>, so that <benefit>.
## 4. Functional Requirements
Numbered list. Each FR is testable.
- **FR-NNN**: The system MUST ...
## 5. Success Criteria
Observable, tech-agnostic metrics.
- **SC-NNN**: ...
## 6. Assumptions & Out of Scope
What we take for granted and what doesn't belong.
## 7. Open Questions
- **[NEEDS CLARIFICATION CL-NNN]** questions that remain open
- **[RESOLVED CL-NNN]** once an answer is verified against code
## 8. Tech Debt Discovered
- **[DEBT-NNN]** findings from reverse engineering
- **[DEBT-NNN] — FIXED YYYY-MM-DD** once the debt is paid down# Plan: <Name>
**Related spec**: ./spec.md
**Type**: Reverse-engineered | Forward
**Status**: Draft | Reviewed | Canonical
**Last synced with code**: YYYY-MM-DD
## 1. Hexagonal Mapping
What lives in what layer (Domain / Application / Infrastructure / Presentation).
## 2. As-Built Flow
ASCII diagram or sequence describing the real flow.
## 3. Ports & Contracts
Interfaces consumed/exposed.
## 4. External Dependencies
URLs, auth, rate limits, timeouts, retry policies.
## 5. Persistence
Tables read/written, key columns, referenced migrations.
## 6. Async / Scheduled Work
Celery tasks (name, schedule, queue, retry), LangGraph nodes.
## 7. Observability
Metrics, logs, audit trail.
## 8. Source Files
Map to real files (with paths relative to `src/`).
## 9. Deviations from Constitution
Known violations of `constitution.md` with justification or debt ticket.spec.mddoes not mention technology. If "HTTP", "SQL", "Celery", "BCRA API" appears — it goes inplan.md.plan.mddoes not invent requirements. It only describes what exists (reverse) or what was decided (forward).- Every tech debt finding is explicitly marked with
[DEBT-NNN]inspec.mdsection 8. - Ambiguities are marked with
[NEEDS CLARIFICATION CL-NNN]— they are opened as questions, not resolved by guessing. - Traceability always present: section 8 of
plan.mdpoints to real files with paths. If a spec cannot point to the code, it is outdated. Last synced with codemust be bumped whenever a spec is touched alongside a code change.- English only across all specs (the i18n strings inside code blocks are implementation artifacts and stay as-is).