feat(corpora): HotpotQA loader for Cat 2c multi-hop cross-validation (#43)

Adds sme/corpora/hotpotqa/ — the Phase-1 multi-hop calibration surface from
upstream M0nkeyFl0wer#43. HotpotQA's
sentence-level annotated supporting facts give Cat 2c (multi-hop retrieval
recall by depth) a public, 1000s-scale corpus with known 2-hop evidence,
extending the construct-validity story beyond jp-realm-v0.1's 10 hand-authored
multi-hop questions.

Mirrors the locomo/longmemeval/beam loader interface exactly:
- HotpotQuestion / HotpotParagraph dataclasses; load_questions(path) iterator;
  materialize_sme_corpus(questions, output_dir) writing per-question vaults
  (each question owns its ~10-paragraph distractor haystack, LongMemEval-style).
- Pinned-subset contract: SUBSET=dev_distractor, SETTING=distractor,
  SUBSET_QUESTION_COUNT=7405, every question 2-hop by construction
  (HOTPOT_MIN_HOPS=2 — the Cat 2c key). type (bridge/comparison) exposed as
  the multi-hop shape; all map to sme_category cat_2c.
- gold paragraphs (those carrying a supporting fact) flagged is_gold;
  expected_sources = gold titles; gold_only materialization for an
  oracle-retrieval upper bound. Tolerant parsing: out-of-range sent_ids and
  malformed supporting_facts entries are skipped, not raised.
- Dataset NOT committed (CC BY-SA 4.0, ~44 MB) — .gitignore + README download
  instructions, matching the locomo/beam convention.

tests/test_hotpotqa_loader.py — 16 schema-fidelity tests against an inline
3-question fixture (bridge / comparison / edge cases), no download needed.
Pins the comparability constants so a silent subset change fails loudly.

scripts/hotpotqa_retrieval_smoke.py — dependency-free lexical (IDF) retrieval
smoke reporting multi-hop recall (both gold paragraphs in top-K). Verified
against the real dev-distractor split (downloaded, reshaped from the HF
mirror to the original JSON format): parses to exactly 7,405 records, all
2-hop with 2 gold paragraphs each; top-5 recall 64% full / 98% partial, with
the expected bridge (>comparison) split that Cat 2c is designed to surface.

docs/sme_spec_v8.md — flip the Cat 2c public-corpus appendix cell from
"HotpotQA integration pending" to the landed loader + smoke numbers; full
Cat 2c cross-val run noted as downstream.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: jphein

docs/sme_spec_v8.md

@@ -1047,7 +1047,7 @@ The appendix is populated incrementally as standard-corpus integration ([#43]) a
 | Category | Hand-authored evidence | Public-corpus evidence | Real-system case studies |
 |---|---|---|---|
 | **Cat 1 — The Lookup** | jp-realm-v0.1 (24 notes, 8 questions) | LongMemEval substrate parity (R@5 = 0.9660 byte-identical to upstream) | — |
-| **Cat 2c — The Stairway (multi-hop)** | jp-realm-v0.1 (10 multi-hop questions at hop depths 1/2/3) | HotpotQA integration pending ([#43]) | — |
+| **Cat 2c — The Stairway (multi-hop)** | jp-realm-v0.1 (10 multi-hop questions at hop depths 1/2/3) | HotpotQA loader landed ([#43] Phase 1): `sme/corpora/hotpotqa/` parses the dev-distractor split (7,405 questions, all 2-hop, 2 gold paragraphs each); lexical retrieval smoke recovers both gold paragraphs for 64% / ≥1 for 98% at top-5. Full Cat 2c cross-val run downstream. | — |
 | **Cat 3 — The Dissonance** | jp-realm-v0.1 (2 contradiction pairs) | — | — |
 | **Cat 4 — The Threshold (Ingestigation)** | jp-realm-v0.1 (3 alias pairs, 5 seeded defects) | MINE integration pending ([#43]) | Content-engine baseline pass 2026-05-05 (canonical-collision bug surfaced + rebuild informed; drafts at [`docs/issue-drafts/2026-05-05-content-engine-feedback/`](issue-drafts/2026-05-05-content-engine-feedback/)) |
 | **Cat 5 — The Missing Room** | jp-realm-v0.1 (1 structural gap between auth_engineering and privacy_research) | MINE / GraphRAG-Bench integration pending ([#43]) | — |

scripts/hotpotqa_retrieval_smoke.py

@@ -0,0 +1,142 @@
+#!/usr/bin/env python
+"""HotpotQA retrieval smoke — Cat 2c multi-hop recall at the smallest slice.
+
+Runs a dependency-free lexical (token-overlap / IDF) retrieval over the
+smallest slice of the HotpotQA dev distractor split and reports multi-hop
+recall: for each question, did the top-K retrieved paragraphs include BOTH
+annotated gold paragraphs? This is the same shape the SME Cat 2c reading
+takes, exercised end-to-end through the loader without needing chromadb or an
+embedding model — a true retrieval smoke, not just a parse check.
+
+The full dev split (hotpot_dev_distractor_v1.json, ~44 MB, CC BY-SA 4.0) is
+gitignored; download per sme/corpora/hotpotqa/README.md. This script reads
+that file if present and otherwise falls back to the inline test fixture so
+it always runs.
+
+Usage:
+    python scripts/hotpotqa_retrieval_smoke.py            # auto-detect data
+    python scripts/hotpotqa_retrieval_smoke.py --n 50     # first 50 questions
+    python scripts/hotpotqa_retrieval_smoke.py --k 5      # top-5 retrieval
+    python scripts/hotpotqa_retrieval_smoke.py --data path/to/hotpot.json
+"""
+from __future__ import annotations
+
+import argparse
+import math
+import re
+from collections import Counter
+from pathlib import Path
+
+from sme.corpora.hotpotqa import HotpotQuestion, load_questions
+
+_WORD_RE = re.compile(r"[a-z0-9]+")
+
+# Default location the README's download instructions write to.
+_DEFAULT_DATA = (
+    Path(__file__).resolve().parent.parent
+    / "sme/corpora/hotpotqa/data/hotpot_dev_distractor_v1.json"
+)
+
+
+def _tokens(text: str) -> list[str]:
+    return _WORD_RE.findall(text.lower())
+
+
+def _idf_retrieve(question: HotpotQuestion, k: int) -> list[str]:
+    """Rank the question's context paragraphs by IDF-weighted token overlap
+    with the question, returning the top-k paragraph titles.
+
+    A minimal lexical retriever — enough to demonstrate the loader feeds a
+    retriever correctly and that multi-hop recall is measurable. The real
+    Cat 2c run uses the daemon/flat adapter; this is the smoke.
+    """
+    paragraphs = question.paragraphs
+    n_docs = len(paragraphs) or 1
+    # document frequency per token across this question's paragraphs
+    df: Counter[str] = Counter()
+    para_tokens: list[set[str]] = []
+    for p in paragraphs:
+        toks = set(_tokens(p.title + " " + p.text))
+        para_tokens.append(toks)
+        for t in toks:
+            df[t] += 1
+
+    q_tokens = _tokens(question.question)
+    scored: list[tuple[float, str]] = []
+    for p, toks in zip(paragraphs, para_tokens):
+        score = 0.0
+        for t in q_tokens:
+            if t in toks:
+                idf = math.log((n_docs + 1) / (df[t] + 0.5))
+                score += idf
+        scored.append((score, p.title))
+    scored.sort(key=lambda s: -s[0])
+    return [title for _, title in scored[:k]]
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--data", type=Path, default=None)
+    ap.add_argument("--n", type=int, default=20, help="questions to score")
+    ap.add_argument("--k", type=int, default=5, help="top-k retrieval")
+    args = ap.parse_args()
+
+    data_path = args.data or (_DEFAULT_DATA if _DEFAULT_DATA.exists() else None)
+    if data_path is None:
+        print(
+            "no dev split found at "
+            f"{_DEFAULT_DATA} — download per sme/corpora/hotpotqa/README.md; "
+            "falling back to the inline fixture for a parse-only smoke."
+        )
+        from tests.test_hotpotqa_loader import FIXTURE  # type: ignore
+        import json
+        import tempfile
+
+        tmp = Path(tempfile.mkdtemp()) / "fixture.json"
+        tmp.write_text(json.dumps(FIXTURE))
+        data_path = tmp
+
+    questions = []
+    for q in load_questions(data_path):
+        questions.append(q)
+        if len(questions) >= args.n:
+            break
+
+    print(f"HotpotQA retrieval smoke — {data_path}")
+    print(f"scoring {len(questions)} questions, top-{args.k} lexical retrieval\n")
+
+    full_recall = 0      # both gold paragraphs in top-k
+    partial_recall = 0   # at least one gold paragraph in top-k
+    by_type: Counter[str] = Counter()
+    by_type_full: Counter[str] = Counter()
+    for q in questions:
+        retrieved = set(_idf_retrieve(q, args.k))
+        gold = set(q.gold_titles)
+        hit = gold & retrieved
+        by_type[q.qtype] += 1
+        if gold and gold <= retrieved:
+            full_recall += 1
+            by_type_full[q.qtype] += 1
+        if hit:
+            partial_recall += 1
+
+    n = len(questions)
+    print(f"  multi-hop recall (both gold paragraphs in top-{args.k}): "
+          f"{full_recall}/{n} = {full_recall / n:.1%}")
+    print(f"  partial recall (>=1 gold paragraph):                    "
+          f"{partial_recall}/{n} = {partial_recall / n:.1%}")
+    print("  by type:")
+    for t in sorted(by_type):
+        print(f"    {t:12} {by_type_full[t]}/{by_type[t]} full multi-hop recall")
+
+    # Smoke gate: lexical retrieval must surface at least one gold paragraph
+    # for the overwhelming majority — if it can't, the loader is feeding the
+    # retriever garbage.
+    ok = n > 0 and partial_recall / n >= 0.9
+    print(f"\n  smoke: {'PASS' if ok else 'FAIL'} "
+          f"(partial recall >= 90%)")
+    return 0 if ok else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

sme/corpora/hotpotqa/.gitignore

@@ -0,0 +1,7 @@
+# HotpotQA upstream JSON — not committed.
+# Dev distractor split (hotpot_dev_distractor_v1.json ~44 MB) and the
+# train split (hotpot_train_v1.1.json ~535 MB). Download per README.md.
+data/
+*.json
+!questions.yaml
+!*.yml

sme/corpora/hotpotqa/README.md

@@ -0,0 +1,124 @@
+# HotpotQA — SME corpus loader
+
+This directory holds the loader and SME-shape conversion for the
+**HotpotQA** benchmark (Yang et al., EMNLP 2018; arXiv 1809.09600).
+The dataset itself is **not committed** to this repo; see the download
+section below.
+
+HotpotQA is the Phase-1 multi-hop calibration surface from the
+standard-corpora integration plan (upstream
+`M0nkeyFl0wer/multipass-structural-memory-eval#43`). It is the public,
+1000s-scale corpus with **sentence-level annotated supporting facts**
+that lets SME demonstrate — not just design — construct validity for
+**Cat 2c** (multi-hop retrieval recall by depth): *"Cat 2c ran against
+HotpotQA's known 2-hop evidence and recovered N of M gold paragraphs."*
+It mirrors the LoCoMo / LongMemEval loaders' interface exactly.
+
+## Pinned subset (the comparability contract)
+
+HotpotQA cross-comparisons are unreliable unless the split and
+retrieval setting are pinned. This loader pins:
+
+| Constant | Value | Meaning |
+|---|---|---|
+| `SUBSET` | `"dev_distractor"` | the `hotpot_dev_distractor_v1.json` split |
+| `SETTING` | `"distractor"` | 10-paragraph haystack (2 gold + 8 distractor) |
+| `SUBSET_QUESTION_COUNT` | `7405` | questions in the dev distractor split |
+| `HOTPOT_MIN_HOPS` | `2` | every question is 2-hop by construction |
+
+**Any reading published from this loader must state the split and
+setting.** The **fullwiki** setting (retrieve over all of Wikipedia) is
+a different, IR-heavy task and is out of scope for a loader. The train
+split (`hotpot_train_v1.1.json`, 90,447 questions) is also loadable —
+pass its path explicitly — but the *pinned* comparability subset is the
+dev distractor split.
+
+## Dataset download
+
+```bash
+mkdir -p sme/corpora/hotpotqa/data
+cd sme/corpora/hotpotqa/data
+# dev distractor split (~44 MB) — the pinned subset
+wget http://curtis.ml.cmu.edu/datasets/hotpot/hotpot_dev_distractor_v1.json
+# optional: train split (~535 MB)
+# wget http://curtis.ml.cmu.edu/datasets/hotpot/hotpot_train_v1.1.json
+```
+
+The `data/` directory is gitignored — keep the upstream JSON local.
+HotpotQA is released under **CC BY-SA 4.0**; redistribution requires
+attribution and share-alike, so the corpus is downloaded per-machine
+rather than vendored here.
+
+## Hop depth (the Cat 2c join)
+
+SME Cat 2c groups questions by `min_hops`. HotpotQA does not annotate an
+explicit integer hop depth, but **every released question is 2-hop by
+construction** (two gold supporting paragraphs), so the loader assigns
+`min_hops = 2` to every record. The `type` field is the qualitative
+multi-hop *shape*:
+
+| `type` | shape | retrieval behavior |
+|---|---|---|
+| `bridge` | sequential 2-hop | resolve a bridge entity in paragraph A, then use it to answer from paragraph B (true chaining) |
+| `comparison` | parallel 2-hop | retrieve a fact from each of two paragraphs and compare them (both must be found, no chaining) |
+
+Both require ≥2 distinct gold paragraphs, hence `min_hops = 2`. A
+deeper-hop corpus (e.g. MuSiQue) would extend the depth axis; HotpotQA
+pins the 2-hop calibration point that jp-realm-v0.1 cannot reach at
+scale.
+
+## Format mapping (HotpotQA → SME)
+
+| HotpotQA field | SME mapping |
+|---|---|
+| `_id` | `question_id` (preserved as the SME question `id` and the per-question vault dir) |
+| `question` | `text` |
+| `answer` | `gold_answer` (QA judge target; `"yes"`/`"no"` for comparison questions) |
+| `type` (`comparison`/`bridge`) | preserved under `hotpotqa.type`; named via `HOTPOT_TYPE_NAMES`; all map to SME `cat_2c` |
+| `level` (`easy`/`medium`/`hard`) | preserved under `hotpotqa.level` |
+| `supporting_facts` (`[[title, sent_id], …]`) | preserved under `hotpotqa.supporting_facts`; gold titles → `expected_sources`; sentence texts via `expected_sources_sentence_level()` |
+| `context` (`[[title, [sentence, …]], …]`) | one markdown file per paragraph under `vault/<question_id>/<title>.md`; gold paragraphs flagged `is_gold: true` |
+| — (assigned) | `min_hops: 2`, `sme_category: cat_2c` |
+
+## Architectural note: per-question vaults (not per-sample)
+
+LoCoMo shares one conversation across all of a sample's questions, so
+its loader writes a vault per *sample*. HotpotQA instead gives each
+question its own ~10-paragraph haystack, so this loader writes a vault
+per *question* (`vault/<question_id>/`) — the same per-question scoping
+as LongMemEval. A cross-validation run loops per question:
+
+```python
+for q in load_questions(dev_distractor_path):
+    adapter.reset()
+    adapter.ingest_corpus_from_dir(vault_dir / q.question_id)
+    result = adapter.query(q.text, n_results=5)
+    sme_score = sme_substring_match(result, q.expected_sources_paragraph_level())
+    # multi-hop recall: did retrieval surface BOTH gold paragraphs?
+    record(q.question_id, sme_score, min_hops=q.min_hops)
+```
+
+`materialize_sme_corpus(..., gold_only=True)` drops the distractors for
+an oracle-retrieval upper bound; the default writes the full distractor
+haystack (the standard setting).
+
+## Status
+
+- `loader.py` — `HotpotQuestion` / `HotpotParagraph` dataclasses,
+  `load_questions(path)` iterator,
+  `materialize_sme_corpus(questions, output_dir)` for per-question vault
+  rendering. Pinned-subset constants exported.
+- `tests/test_hotpotqa_loader.py` — schema-fidelity tests against an
+  inline fixture (no download needed).
+- **Pending (downstream):** the Cat 2c cross-validation run against the
+  daemon. The loader is the prerequisite; the run is a separate task.
+
+## Citation
+
+Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov,
+R., & Manning, C. D. (2018). *HotpotQA: A Dataset for Diverse,
+Explainable Multi-hop Question Answering.* EMNLP 2018. arXiv:1809.09600.
+
+Upstream repo: https://github.com/hotpotqa/hotpot
+Project page: https://hotpotqa.github.io/
+License: CC BY-SA 4.0

sme/corpora/hotpotqa/__init__.py

@@ -0,0 +1,52 @@
+"""HotpotQA corpus loader for SME cross-validation.
+
+HotpotQA (Yang et al., EMNLP 2018; arXiv 1809.09600) is a Wikipedia-based
+multi-hop question-answering benchmark with sentence-level annotated
+supporting facts. This package loads the released benchmark JSON and produces
+SME-shape question records so SME's **Cat 2c** (multi-hop retrieval recall by
+depth) reading can be cross-validated against a public, 1000s-scale corpus
+with known multi-hop evidence — the construct-validity demonstration the
+hand-authored corpora cannot give (upstream
+M0nkeyFl0wer/multipass-structural-memory-eval#43, Phase 1).
+
+PINNED SUBSET: ``dev_distractor`` — ``hotpot_dev_distractor_v1.json``, the
+**distractor** setting (10-paragraph haystack: 2 gold + 8 distractor),
+**7405** questions. Every question is 2-hop by construction, so the loader
+assigns ``min_hops = 2`` to every record (the Cat 2c key) and exposes the
+``type`` field (comparison vs bridge) as the multi-hop shape. See loader.py's
+module docstring for the comparability contract and the hop-depth rationale.
+
+The dataset itself is NOT committed to this repo. Download with:
+
+    mkdir -p sme/corpora/hotpotqa/data
+    cd sme/corpora/hotpotqa/data
+    wget http://curtis.ml.cmu.edu/datasets/hotpot/hotpot_dev_distractor_v1.json
+
+See README.md for details and the SME mapping table.
+"""
+
+from sme.corpora.hotpotqa.loader import (
+    HOTPOT_MIN_HOPS,
+    HOTPOT_SME_CATEGORY,
+    HOTPOT_TYPE_NAMES,
+    SETTING,
+    SUBSET,
+    SUBSET_QUESTION_COUNT,
+    HotpotParagraph,
+    HotpotQuestion,
+    load_questions,
+    materialize_sme_corpus,
+)
+
+__all__ = [
+    "HOTPOT_MIN_HOPS",
+    "HOTPOT_SME_CATEGORY",
+    "HOTPOT_TYPE_NAMES",
+    "SETTING",
+    "SUBSET",
+    "SUBSET_QUESTION_COUNT",
+    "HotpotParagraph",
+    "HotpotQuestion",
+    "load_questions",
+    "materialize_sme_corpus",
+]