feat(corpus-doctor): synthetic defect injection + verification harness (#27 first slice)

[jphein]

What

First slice of upstream M0nkeyFl0wer/multipass-structural-memory-eval#27 — corpus-doctor, a synthetic-defect-injection tool that calibrates SME's structural probes.

SME's public corpora are integrity-clean by construction, which leaves Cat 4 (The Threshold) and Cat 5 (The Missing Room) un-calibrated: they always read clean on clean input, so a passing reading proves nothing about detector sensitivity. corpus-doctor closes the loop — inject a KNOWN defect, emit a PROV-O manifest, run the cat, assert it recovers what was injected.

Pathologies (3 of 6, no new dependency)

Pathology	Detector	Signal
duplicate_evidence	Cat 4a	canonical_collisions (+1 each) — clones entities under fresh IDs but identical name+type
orphan_inflation	Cat 5	isolated_nodes (+1 each) — strips all edges off sampled entities
monoculture_edge_type	Cat 4c	dominant_edge_type_fraction (↑) — collapses edges onto one type (defaults to amplifying the existing dominant)

Each injector is a pure (entities, edges) -> DoctorResult transform over the core dataclasses. Injection is deterministic given seed; severity (0..1) scales the count/fraction.

Design notes

• PROV-O manifest (defects.jsonl): one record per defect with prov:activity=inject_defect, prov:wasAttributedTo=corpus-doctor/0.1, and an expect block stating the category/field/delta the cat should move — reusing the repo's existing no-dep PROV-O-JSON convention (docs/industry_standards_integration.md). Round-trips via write_manifest/load_manifest.
• Verification harness imports the REAL Cat 4/5 scorers (no reimplemented detection logic), so a passing test proves the shipped detector is sensitive.
• Collateral isolation: orphaning a node can strand a degree-1 neighbour, so Cat 5 may report more isolates than defects. The harness caps recovery at 100% (the detector is right about the collateral, not penalised) and surfaces it as a note. This was caught and root-caused during dev, not papered over.
• monoculture default: amplifies the corpus's existing dominant edge type. An early version hard-defaulted to a fixed string RELATED, which on good-dog (dominant = mentions) actually lowered the dominant fraction by redistributing share — a real bug the auto-detect default fixes.

CLI

```bash
sme-eval corpus-doctor # verify all 3 on good-dog baseline
sme-eval corpus-doctor --pathology orphan_inflation --severity 0.5 --out-dir DIR
sme-eval corpus-doctor --from-adapter mempalace --db PATH --json report.json
```

Non-zero exit on any undetected pathology → doubles as a CI calibration gate.

Validation

All three pathologies recover 100% on the in-tree good-dog corpus (164 edges):

```
duplicate_evidence canonical_collisions clean=0 dirty=29 obs=+29 expected +29 recovery=100% DETECTED
monoculture_edge_type dominant_edge_type_fraction clean=0.348 dirty=0.543 obs=+0.195 expected ↑ recovery=100% DETECTED
orphan_inflation isolated_nodes clean=0 dirty=34 obs=+34 expected +29 recovery=100% DETECTED
```

Tests

29 new tests in tests/test_corpus_doctor.py (inject → detect round-trip, exact-count assertions, determinism, severity scaling, manifest round-trip, end-to-end on the real corpus). ruff check clean. Full suite green except 2 pre-existing live-network hindsight failures (confirmed independent of this change).

Deferred (flagged, not built — first slice scope)

PATHOLOGY_BACKLOG in code names the remaining issue-#27 pathologies: zipfian_degree, hotspot_entity, stale_facts, and phantom_edge (the last blocked on the Cat phantom-edge detector, upstream #4). TextAttack remains an opt-in [text-perturb] follow-up.

Refs upstream M0nkeyFl0wer/multipass-structural-memory-eval#27.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces corpus-doctor, a synthetic defect injection tool designed to calibrate and stress-test structural memory evaluation detectors (specifically Cat 4 and Cat 5). It implements three initial pathologies (duplicate_evidence, orphan_inflation, and monoculture_edge_type), along with a verification harness, CLI integration, and comprehensive tests. The review feedback suggests two key improvements: first, ensuring that cloned entities in sme/corpus_doctor.py preserve the original entity's embedding to prevent issues with downstream components; second, including the properties field when serializing the dirtied snapshot to snapshot.json in sme/cli.py to avoid losing crucial metadata injected by the tool.

[gemini-code-assist]

The cloned entity clone is instantiated without passing embedding=src.embedding. This means the cloned entity's embedding defaults to None. If downstream components or other categories rely on entity embeddings (e.g., for hybrid search or vector-based evaluation), they might fail or behave unexpectedly because the cloned entity lacks an embedding. We should preserve the embedding by reference, just like _clone_snapshot does.

Suggested change

	clone = Entity(
	id=clone_id,
	name=src.name,
	entity_type=src.entity_type,
	properties={
	**copy.deepcopy(src.properties),
	"_corpus_doctor": "duplicate_evidence",
	"_dupe_of": src.id,
	},
	)
	clone = Entity(
	id=clone_id,
	name=src.name,
	entity_type=src.entity_type,
	properties={
	**copy.deepcopy(src.properties),
	"_corpus_doctor": "duplicate_evidence",
	"_dupe_of": src.id,
	},
	embedding=src.embedding,
	)

[gemini-code-assist]

When writing the dirtied snapshot to snapshot.json, the properties field of both entities and edges is discarded. However, the synthetic defect injectors store crucial metadata in properties (such as _corpus_doctor, _dupe_of, _original_edge_type, etc.). Discarding properties makes the serialized snapshot incomplete and less useful for debugging or downstream tools. We should include the properties field in the serialized JSON.

            "entities": [
                {
                    "id": e.id,
                    "name": e.name,
                    "entity_type": e.entity_type,
                    "properties": e.properties,
                }
                for e in dirty.entities
            ],
            "edges": [
                {
                    "source_id": e.source_id,
                    "target_id": e.target_id,
                    "edge_type": e.edge_type,
                    "properties": e.properties,
                }
                for e in dirty.edges
            ],

[jphein]

Closing as duplicate of #250 (already merged). Two agents independently built corpus-doctor for #27; #250 (the sme/corpus_doctor/ package — deterministic injector + manifest-as-ground-truth + recall/Δprecision harness, 17 tests) was merged first and is the chosen architecture. This branch's single-file layout + CLI integration would collide. If the sme/cli.py corpus-doctor subcommand from this branch is wanted, it can be a small follow-up on top of the merged package. No work lost — the capability is on main.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.