Unify reproducible benchmark result storage

[janpower]

Summary

• add a shared result store for retrieval, extraction, and LLM benchmarks
• preserve collision-free run directories with manifests, checksums, canonical term/case artifacts, and legacy compatibility
• add CSC/GSC LLM result support, checkpoint identity validation, recursive artifact discovery, and comprehensive tests
• bind checkpoint reuse to dataset contents and accounting configuration
• retain deprecated LLM output paths during migration and record non-evaluable retrieval cases explicitly

Motivation

Benchmark outputs previously used overlapping flat paths and inconsistent artifact formats. This made repeated runs, checkpoint resume, cross-model comparison, and downstream manuscript analysis difficult to reproduce reliably.

This PR contains only reusable Phentrieve benchmark infrastructure, tests, and technical user documentation. Manuscript-specific analyses, run data, logs, figures, and planning documents are intentionally excluded.

Validation

• make check
• make typecheck-fast
• uv sync --all-extras --frozen
• make test (2023 passed, 48 skipped)
• focused benchmark regression suite (85 passed)

[janpower]

Independent review findings addressed in ec70549: dataset-content and accounting fingerprints now gate checkpoint reuse; deprecated output paths remain supported with warnings; retrieval cases without gold IDs are recorded as not_evaluable. Full suite: 2023 passed, 48 skipped; check and typecheck pass.

[berntpopp]

Deep review — full-stack + RAG/LLM lens, plus an independent Codex (gpt-5.5, xhigh) pass

Strong PR: the run-directory + manifest + checksum model is the right shape, _atomic_text_write is properly atomic, the CSC/GSC support is clean, and binding checkpoint reuse to the resolved provider/model (rather than raw CLI input) is a subtle call that's exactly right.

I found six defects and fixed them on the branch. Both reviews converged on the same top-two independently, which is a good signal they're real. Each is covered by a regression test.

Fixed

1. Overwritten runs inherited artifacts they never produced (high — this defeats the PR's core promise)
create_run_layout(overwrite=True) reused the directory without clearing it, and write_manifest decides what to advertise purely from Path.exists(). So run 2 published run 1's files as its own. Reproduced:

run 1 files: ['diagnostics/chunks.jsonl', 'manifest.json', 'predictions/doc_A.json', 'terms.jsonl']
run 2 (--overwrite, no diagnostics this time):
  manifest artifacts: ['term_results', 'chunk_diagnostics']   <-- claims chunks it never wrote
  chunk_diagnostics content: {"chunk_id":0,"run":1}           <-- run 1's data

Now the run directory is cleared on overwrite, preserving only checkpoint.json — the LLM benchmark reuses a directory precisely to resume from it, and that checkpoint's identity is validated separately. RunLayout owns the checkpoint filename now instead of it being a duplicated literal.

2. benchmark compare listed extraction and LLM runs as retrieval models (high)
All three benchmark types write a summary role into the shared results root, and discover_artifacts was type-blind. Every extraction/LLM run added a junk row to the dense-retrieval table:

      Model   Dataset  Run ID   MRR (Dense)
BAAI/bge-m3 pheno_test   ret1          0.71
    Unknown   GSC_plus   ext1          0.00   <-- extraction run
    Unknown        CSC   llm1          0.00   <-- LLM run

discover_artifacts takes an optional benchmark_type filter; both retrieval loaders set it. (The guide worked around this with --summaries-dir results/retrieval; the default is now correct, so I updated it.)

3. Checkpoint identity didn't cover the prompt templates (high — RAG-specific)
Identity recorded prompt_templates_dir but not its contents. The prompt is an experimental variable: edit a template in place, resume the same --run-id, and doc 1's old-prompt output silently merges with doc 2's new-prompt output into one set of metrics. The PR already hashes the dataset — prompts deserve the same treatment. Now hashed (packaged + effective user set, which is ~/.phentrieve/prompts even without the flag). Checkpoints predating the key stay resumable rather than being invalidated on upgrade; ones written from now on are compared strictly.

4. LLM manifests always claimed status: complete (high)
run_llm_benchmark returns status="completed" for any run that finishes its loop — including one where every document failed — and the CLI mapped anything != failed to complete. Status is now derived from per-case outcomes (complete/partial/failed), as the retrieval runner already did, with failed/complete counts in the manifest.

5. cases.jsonl mixed two key schemes (medium)
Evaluable/failed cases used 0-based case_id (what terms.jsonl joins on); non-evaluable cases used 1-based case_index and no case_id — so a downstream join silently dropped exactly the rows the manifest counts under not_evaluable. Unified on case_id.

6. Path leaked into public result dicts (medium)
run_evaluation() returned run_dir as a Path (the LLM payload already returned str), and swapping glob.glob → discover_artifacts flipped result_analyzer's injected file_path from str to Path. Either breaks a caller that JSON-serializes the result. Both now str.

Also: the distance→similarity fallback open-coded 1.0 - d while the retriever converts through calculate_similarity (which clamps to [0,1]); an unclamped score feeds passes_threshold in the chunk diagnostics. Switched to the shared helper. And traces/ are always written, not gated on --capture-phase1-debug as the guide claimed — doc corrected.

Checked, no finding

Retrieval scoring semantics are sound: the store is ChromaDB cosine, DenseRetriever returns similarities, and 1 - d matches the existing aggregation.py convention. discover_artifacts deliberately ignores stray flat *_summary.json once canonical runs exist — that's an explicit, tested decision, so I left it alone.

Also folded in

Dependabot #307 (vite 8.1.4) and #308 (8 frontend deps). Both edit the same block, so package.json is the union of the two and the lockfile was regenerated from it. The prettier 3.8.4 → 3.9.5 bump reformatted nothing.

Versions

chore(release): 0.26.0 (CLI 0.26.0 / API 0.17.0 / Frontend 0.17.1) — CLI minor for the result store and new flags, frontend patch for the dependency bumps. API is untouched by this PR and stays at 0.17.0.

Verified locally: make ci-local green (Python 2030 passed / 78% coverage; frontend 326 passed, lint + format + build), make security-python clean (6 pre-existing LOW findings, all in untouched files).

🤖 Generated with Claude Code