feat: add point-in-time market data catalog

Add an immutable SQLite-backed provenance catalog, a bounded read-only Polymarket Gamma adapter, CLI commands, adversarial tests, and V0.11 traceability updates.

Co-Authored-By: Codex <noreply@openai.com>

Author: hihihhi

README.md

@@ -4,19 +4,21 @@ QRAE is a local-first, human-gated quantitative research operating system. The r
 
 ## Current truth
 
-Source release candidate: `0.10.0`. Verify any hosted surface separately; a deployment timestamp or endpoint response never proves that the local research kernel ran.
+Source release candidate: `0.11.0`. Verify any hosted surface separately; a deployment timestamp or endpoint response never proves that the local research kernel ran.
 
 Implemented and tested:
 
 - strict, hash-bound research work orders;
 - point-in-time CSV price contracts and data-quality quarantine;
+- a durable SQLite point-in-time catalog with immutable hash-addressed raw/normalized objects, provenance, idempotency, tamper checks, and non-regressing latest pointers;
+- one bounded, unauthenticated, read-only Polymarket Gamma market-metadata adapter capped at `E0`;
 - a delayed-fill lagged-momentum baseline with registered entry/liquidation costs and chronological train/validation/test splits;
 - immutable run bundles, atomic latest pointers, and tamper-evident state logs;
 - metric/accounting recomputation, evidence ceilings, claim ledgers, decision records, compact reports, and knowledge records;
 - an optional local Codex broker with fixed tasks, an isolated read-only permission profile, bounded output, no state-promotion authority, and deterministic no-Codex fallback;
 - legacy read-only Vercel dashboard/API views; deployment content may lag this branch and is not execution evidence.
 
-Not yet implemented: production market adapters, remote storage/batch adapters, walk-forward and nested model selection, event-driven replay, portfolio/risk engines, ML challengers, a complete Obsidian brain, allocator-grade report generation, authenticated web review, or any paper/live execution system.
+Not yet implemented: historical/streaming production market adapters, remote storage/batch adapters, walk-forward and nested model selection, event-driven replay, portfolio/risk engines, ML challengers, a complete Obsidian brain, allocator-grade report generation, authenticated web review, or any paper/live execution system.
 
 The committed dashboard artifacts and legacy synthetic tournament are demonstrations, not evidence that those missing systems ran.
 
@@ -36,6 +38,15 @@ python -m qrae.cli verify --run-dir .quantos/runs/<run-id>
 
 The bundled dataset is synthetic and its work order is capped at `E0`. It proves mechanics and artifact integrity only.
 
+Fetch and independently rehash one bounded public metadata snapshot:
+
+```text
+python -m qrae.cli polymarket-fetch --catalog-root .quantos/catalog --request-id pm-YYYYMMDD-001 --limit 100
+python -m qrae.cli catalog-verify --catalog-root .quantos/catalog
+```
+
+The adapter performs no trading, authentication, pagination, or historical reconstruction. “Observed at retrieval” data remains `E0` and cannot support a historical-alpha claim.
+
 ## Local Codex review
 
 Codex is optional and never owns metrics or state transitions. Prepare a fixed review task from a verified run:
@@ -63,7 +74,7 @@ To invoke the locally authenticated Codex CLI, repeat with `--enable-codex`. The
 ## Repository map
 
 ```text
-src/qrae/                    deterministic contracts, kernel, state, artifacts, baseline, Codex broker
+src/qrae/                    kernel plus PIT catalog, public-data adapter, and Codex broker
 tests/                       positive and negative deterministic tests
 examples/price_baseline/     synthetic E0 work order and price fixture
 scripts/qrae_run_cycle.py    compatibility wrapper for the local kernel

api/_qrae.js

@@ -13,13 +13,13 @@ function readJson(relativePath, fallback) {
 }
 
 const runtimeConfig = readJson('configs/qrae_runtime.json', {});
-const VERSION = runtimeConfig.version || '0.10.0';
+const VERSION = runtimeConfig.version || '0.11.0';
 const GENERATED_AT = '2026-07-05T00:00:00.000Z';
 
 const qraeRuntime = {
   system: 'QRAE-RD',
   version: VERSION,
-  mode: runtimeConfig.mode || 'v0.10-local-first-e0-kernel',
+  mode: runtimeConfig.mode || 'v0.11-local-first-e0-kernel-catalog',
   environment: process.env.QRAE_ENVIRONMENT || process.env.VERCEL_ENV || 'local',
   liveTradingEnabled: false,
   automatedLlmEnabled: false,

api/quality/gates.js

@@ -4,7 +4,7 @@ module.exports = function handler(req, res) {
   if (!methodGuard(req, res)) return;
   const gates = artifactJson('configs/qrae_quality_gates.json', {
     system: 'QRAE-RD',
-    version: '0.10.0',
+    version: '0.11.0',
     quality_guard_status: 'missing',
     test_cases: [],
     fatal_failures: [],

api/requirements/traceability.js

@@ -4,7 +4,7 @@ module.exports = function handler(req, res) {
   if (!methodGuard(req, res)) return;
   const traceability = artifactJson('configs/qrae_requirements_traceability.json', {
     system: 'QRAE-RD',
-    version: '0.10.0',
+    version: '0.11.0',
     traceability_status: 'missing',
     requirements: [],
   });

api/trials/parallel-latest.js

@@ -4,7 +4,7 @@ module.exports = function handler(req, res) {
   if (!methodGuard(req, res)) return;
   const trial = artifactJson('configs/qrae_parallel_trial_001.json', {
     system: 'QRAE-RD',
-    version: '0.10.0',
+    version: '0.11.0',
     trial_id: 'missing',
     instances: [],
     aggregate_feedback: {},

configs/qrae_parallel_trial_001.json

@@ -1,6 +1,6 @@
 {
   "system": "QRAE-RD",
-  "version": "0.10.0",
+  "version": "0.11.0",
   "trial_id": "parallel_trial_001",
   "trial_mode": "three-instance-prod-orchestration-dry-run",
   "generated_at": "2026-07-05T19:00:00Z",

configs/qrae_quality_gates.json

@@ -1,6 +1,6 @@
 {
   "system": "QRAE-RD",
-  "version": "0.10.0",
+  "version": "0.11.0",
   "quality_guard_status": "active",
   "purpose": "Outcome-quality guardrail and test-case suite for QRAE-RD research outputs.",
   "decision_labels": ["AUTO_REJECT", "AUTO_REVISE", "AUTO_DEEPEN", "HUMAN_REVIEW_REQUIRED", "PAPER_TRADE_CANDIDATE"],

configs/qrae_requirements_traceability.json

@@ -1,6 +1,6 @@
 {
   "system": "QRAE-RD",
-  "version": "0.10.0",
+  "version": "0.11.0",
   "traceability_status": "active",
   "generated_from": "rebaselined against executable repository evidence on 2026-07-11",
   "status_legend": {
@@ -302,13 +302,17 @@
       "id": "DATA-001",
       "area": "data_pipeline",
       "requirement": "Handle data downloading/fetching/pulling, cleansing, categorizing, labeling, validation, and storage/database access.",
-      "status": "documented",
+      "status": "partial",
       "implemented_in": [
-        "docs/ARCHITECTURE.md",
-        "api/cron/daily-research.js",
-        "configs/qrae_runtime.json"
+        "src/qrae/data_catalog.py",
+        "src/qrae/polymarket.py",
+        "tests/test_data_catalog.py",
+        "tests/test_polymarket_adapter.py"
       ],
-      "notes": "V0 provides contracts, quality validator, and cron trigger shell. Durable DB/Blob persistence is V1 setup."
+      "notes": "V0.11 implements one bounded read-only public metadata fetch, normalization, durable PIT catalog, and immutable object storage. Broader sources, labels, pagination, and production ingestion remain unimplemented.",
+      "verification": [
+        "python -m pytest -q tests/test_data_catalog.py tests/test_polymarket_adapter.py"
+      ]
     },
     {
       "id": "DATA-002",
@@ -329,11 +333,16 @@
       "id": "DATA-003",
       "area": "storage_policy",
       "requirement": "Use DB/object storage for artifacts and avoid putting heavy tick data in ChatGPT or ordinary Postgres rows.",
-      "status": "documented",
+      "status": "partial",
       "implemented_in": [
+        "src/qrae/data_catalog.py",
+        "tests/test_data_catalog.py",
         "docs/ARCHITECTURE.md"
       ],
-      "notes": "V0 stores artifacts in repo for deployability. V1 should move mutable data/artifacts to Postgres + Blob/S3/R2."
+      "notes": "V0.11 separates SQLite metadata from immutable hash-addressed raw/normalized objects locally. Remote object storage and heavy tick-data tiers remain future work.",
+      "verification": [
+        "python -m pytest -q tests/test_data_catalog.py"
+      ]
     },
     {
       "id": "DATA-004",

configs/qrae_runtime.json

@@ -1,7 +1,7 @@
 {
   "system": "qrae-rd",
-  "version": "0.10.0",
-  "mode": "v0.10-local-first-e0-kernel",
+  "version": "0.11.0",
+  "mode": "v0.11-local-first-e0-kernel-catalog",
   "liveTradingEnabled": false,
   "automatedLlmEnabled": false,
   "directorModeWhenAutomated": "disabled",

docs/ARCHITECTURE.md

@@ -74,7 +74,7 @@ The current kernel implements one generic price-series baseline:
 8. create a claim ledger, decision proposal, knowledge record, report, manifest, and hash-chained state log;
 9. stop at `HUMAN_REVIEW` or fail closed at `QUARANTINED`.
 
-This V0 slice is capped at E0 mechanics evidence. E1 remains blocked until authenticated provenance and a registered market adapter exist. It does not model realistic fills, capacity, borrow, funding, settlement, portfolio constraints, or market-specific events.
+The catalog slice adds SQLite metadata, immutable content-addressed raw/normalized objects, point-in-time resolution, provenance hashes, and a bounded read-only Polymarket Gamma metadata adapter. Because Gamma availability is only observed when fetched and API terms remain unverified, these snapshots are capped at E0. E1 still requires a licensed source with defensible historical availability timestamps plus a compatible backtest contract. No current slice models realistic fills, capacity, borrow, funding, settlement, portfolio constraints, or market-specific events.
 
 ## State machine
 
@@ -101,8 +101,8 @@ Codex output is draft evidence with `requested_transition=NONE`. Disabled or una
 
 ## Planned planes
 
-1. **Kernel:** typed contracts, SQLite catalog, budgets, retries, approvals, and recovery.
-2. **Data:** source registry; raw/normalized/feature layers; DuckDB/Polars/Parquet; point-in-time timestamps and licenses.
+1. **Kernel:** extend the initial SQLite catalog with budgets, retries, approvals, and recovery.
+2. **Data:** extend the first Gamma metadata adapter into a source registry; raw/normalized/feature layers; DuckDB/Polars/Parquet; licensed historical point-in-time timestamps.
 3. **Research:** hypothesis tournament, cheap falsification, WFO, replay, portfolio construction, independent replication.
 4. **Knowledge:** Obsidian-compatible objects, full-text search, typed provenance graph, negative results and lessons.
 5. **Model:** simple baselines before regularized, boosted-tree, neural, ensemble, and eventually simulator-gated RL challengers.