Connect the co-scientist to BERIL `compendium`

[caufieldjh]

See kbaseincubator/BERIL-research-observatory#289

BERIL/BERDL already includes an LLM-generated knowledge layer: atlas.
compendium is intended to serve a similar function but with a fully deterministic build process and sentence-level provenance rather than AI-authored pages alone with page-level evidence.

This would be a great knowledge source for the EcoSIM co-scientist to work with.

[github-actions]

Issue Summary

This is a focused follow-on to #24 (Connect to BERIL Research Observatory), zooming in on a specific new component: compendium — a deterministic, statement-card-centered knowledge synthesis wiki being introduced in BERIL PR #289 (DRAFT, by Dileep Kishore, +12,253 lines).

---

What Is compendium?

compendium is the proposed successor to BERIL's existing atlas/ knowledge layer. Both encode the same scientific epistemics — claims, conflicts, hypotheses, evidence, directions — but they are architecturally inverted:

Dimension	atlas (existing)	compendium (PR #289)
Provenance	Page-level (source_projects, prose evidence)	Sentence-level: every claim carries inline [stmt:<id>; <project>] traceable to a source span
Determinism	Render layer deterministic; content + edges LLM-authored → not reproducible	Python core never calls an LLM; statement graph + page plans are order-independent and reproducible
Drift resistance	Edges may not exist in the corpus (hand-authored)	Edges derived deterministically from extracted statements
Schema	Lint not in CI	Single LinkML source of truth; quality gates enforced by tests
Review workflow	last_reviewed + reviewer briefs	Deterministic review-queue ranking (centrality / conflict / low-confidence)
Output	Live FastAPI/Jinja UI + Cytoscape graph	Connected Markdown wiki (compendium/wiki/)

The core principle: the deterministic Python core never calls an LLM. LLM extraction and prose synthesis happen only inside skills, which publish validated, cached artifacts — never live calls on the render path.

Build pipeline

audit/context-pack → validate cards → statement-graph → plan-pages → wiki-contexts
  → page-artifact (LLM-authored, validated) → render-markdown → quality-synthesis/review-queue

Statement card types

finding | claim | caveat | conflict | hypothesis | opportunity | direction | method_note | derived_product

Each card has a tier: asserted → grounded → reviewed → conflict → retracted, with polarity-aware typed edges and conflict nodes.

Current status

• DRAFT PR; proof-of-concept uses a 3-project ADP1 tracer fixture (acinetobacter_adp1_explorer, adp1_deletion_phenotypes, adp1_triple_essentiality) — not the full corpus
• 87 passing tests covering audit, statement cards, graph assembly, page planning, Markdown publishing, review queue
• Full-corpus ingestion is follow-up work

---

Why This Is Valuable for EcoSIM Co-Scientist

The EcoSIM co-scientist already has:

• BERVO ontology (2,076 terms mapping EcoSIM variables to human-readable concepts)
• 50-paper meta-analysis corpus on warming/nitrogen cycling
• 626-row N measurements database

compendium's architecture maps directly onto the co-scientist's needs:

1. The 50 warming/nitrogen papers as a compendium project corpus: Each paper's findings (e.g., "warming increased net N mineralization by 21%") could be ingested as statement cards, yielding a sentence-level provenance-tracked KG over the meta-analysis literature — far more structured than the current XLSX/TSV database.

2. BERVO as the entity/ontology grounding layer: Statement cards can carry typed links to named entities. BERVO term IDs (e.g., BERVO:0000001) could ground the entities in compendium statement cards — creating a bidirectional bridge between the literature synthesis and EcoSIM's parameter space.

3. From claim to calibration: The opportunity and direction card types could encode parameter calibration signals — e.g., "warming by 2°C increases gross N mineralization rate, suggesting upward adjustment of EcoSIM mineralization parameters" — with full provenance back to the source paper sentence.

4. Determinism = reproducibility for scientific modeling: In modeling workflows, knowing exactly which literature statements informed a parameter choice (and being able to re-derive that from scratch) is exactly what scientific rigor demands. Compendium's deterministic build is a much better fit than LLM-authored atlas pages.

---

Relationship to Issue #24

Issue	Scope
#24	Broad BERIL/BERDL integration (data lakehouse, Spark SQL, shared skills)
#25 (this issue)	Specifically compendium: the deterministic, sentence-level knowledge synthesis layer

These are complementary. #24 is about data and compute access; #25 is about structured scientific knowledge synthesis. An ideal integration would use BERDL for data retrieval (issue #24) while using compendium-style statement cards to track what the co-scientist has learned from the literature (this issue).

---

Suggested Next Steps

• Monitor BERIL PR #289 for merge — compendium is still DRAFT; watch for when it lands
• Prototype ingesting the 50 warming/nitrogen papers as a compendium-style project (even before upstream merge, the architecture is clear enough to build against)
• Define BERVO-grounded entity types for use in statement card entities fields — this is the natural extension of issue Align the ecosim input with BERVO #8 (Align ecosim input with BERVO)
• Evaluate whether the kg-ingest-project skill can be reused directly or needs an EcoSIM/climate-science adaptation (current fixture is microbiology-focused)
• Consider contributing an ecosystem/biogeochemistry project fixture upstream to the BERIL compendium tracer suite

Auto-summarized by ecosim-co-scientist summarization agent.