docs: strengthen lost-in-conversation bench comparison

Author: yha9806

docs/product/lost-in-conversation-writing-control-design.md

@@ -147,7 +147,7 @@ Academic Writing Toolkit turns scattered multi-turn writing intent into thesis-c
 
 ## Implementation Pointer
 
-The first implementation should use the public-safe fixture in `examples/lost-in-conversation-bench/` and the structural checker `scripts/check_lost_in_conversation_bench.py`. The treatment packet should also validate with:
+The first implementation uses the public-safe fixture in `examples/lost-in-conversation-bench/` and the structural checker `scripts/check_lost_in_conversation_bench.py`. The fixture includes actual edited outputs for Baseline A, Baseline B, and Treatment, plus `comparison_report.md` for inline human review. The treatment packet should also validate with:
 
 ```sh
 python3 .claude/skills/thesis-control/scripts/check_thesis_control.py examples/lost-in-conversation-bench/treatment --strict --json

examples/lost-in-conversation-bench/README.md

@@ -10,6 +10,16 @@ The bench compares three workflows on the same desensitised thesis-style section
 
 The fixture is not a model benchmark and does not claim scientific validity. It is a product-control check: can the author inspect what changed, decide whether the edit stayed inside scope, and choose accept, partial accept, revise, or rollback without rereading a long chat history?
 
+Review these files in order:
+
+1. [`chapters/desensitized_section.md`](chapters/desensitized_section.md) records the original section.
+2. [`requirements/multi_turn_requirements.md`](requirements/multi_turn_requirements.md) records the sharded multi-turn requirements.
+3. [`baselines/baseline_a_edited_section.md`](baselines/baseline_a_edited_section.md) shows a normal multi-turn edit with claim and evidence-boundary drift.
+4. [`baselines/baseline_b_edited_section.md`](baselines/baseline_b_edited_section.md) shows the consolidated-prompt output.
+5. [`treatment/edited_section.md`](treatment/edited_section.md) shows the `/thesis-control` bounded edit.
+6. [`comparison_report.md`](comparison_report.md) compares all three workflows across the control metrics.
+7. [`treatment/thesis_control/`](treatment/thesis_control/) stores the spine card, edit contract, and drift audit.
+
 Run from the repository root:
 
 ```bash

examples/lost-in-conversation-bench/baselines/baseline_a_edited_section.md

@@ -0,0 +1,7 @@
+# Section 2.3: Evidence Boundaries In Assisted Revision
+
+AI-assisted thesis revision can become unreliable when a writer depends on multi-turn conversation to correct the direction of a draft. A paragraph may become smoother after several rounds of clarification, but the assistant can still lose track of the original argument and produce claims that go beyond the available evidence. This shows that AI-assisted writing needs explicit controls because ordinary conversational revision is not enough to prevent drift.
+
+The section draws on two source notes. Source A shows that notes, claims, and revision decisions should be kept in separate records. Source B shows that multi-turn assistant use can fold later clarifications into earlier assumptions. Together, these sources show that AI writing tools can become unsafe when they are used across several turns without visible boundaries. They therefore support the need for a workflow that prevents model unreliability from entering thesis prose.
+
+The next subsection can describe how this control workflow should be implemented. The important point here is that fluency can hide deeper problems in the assistant's reasoning. A revised paragraph should therefore be treated as unreliable unless the workflow can prove that the author's claim, evidence boundary, and neighbouring section relationship have all been preserved.

examples/lost-in-conversation-bench/baselines/baseline_b_edited_section.md

@@ -0,0 +1,7 @@
+# Section 2.3: Evidence Boundaries In Assisted Revision
+
+This section argues that AI-assisted thesis revision is useful only when the author can inspect how local edits affect the section's claim boundary. A smoother paragraph can still blur the distinction between what the sources show and what the author infers from them. The section therefore makes a bounded claim: it does not argue that all AI-assisted writing is unreliable, but that revision workflows need explicit controls around claims, caveats, and evidence use.
+
+The section draws on two source notes. Source A describes a writing-support workflow where notes, claims, and revision decisions are kept as separate records. Source B discusses multi-turn assistant use and warns that later clarifications can be folded into earlier assumptions. Together, these sources support the need for visible edit boundaries when thesis writers use AI across several turns. They do not support a universal model-safety claim or a claim that any particular model is unsafe for all writing tasks.
+
+The next subsection discusses implementation, so this section should not introduce product architecture in detail. Its role is to motivate the control problem and define what must not drift during revision. The key caveat is that fluency is not enough. The author must still be able to check whether a smoother paragraph preserves the intended claim, the evidence boundary, and the relationship to neighbouring sections.

examples/lost-in-conversation-bench/comparison_report.md

@@ -0,0 +1,29 @@
+# Lost-in-Conversation Bench Comparison Report
+
+## Summary
+
+This report compares three outputs from the same desensitised thesis-style section and the same revision pressure. Baseline A represents ordinary multi-turn chat editing. Baseline B represents a consolidated single-turn prompt. Treatment represents `/thesis-control`, where the edit is bound to a spine card, edit contract, and drift audit before acceptance.
+
+The comparison is not a model benchmark. It is an author-control review: can the author inspect what changed, decide whether the edit stayed inside scope, and choose accept, partial accept, revise, or rollback without reconstructing the whole conversation?
+
+## Three-Way Metric Comparison
+
+| Metric | Baseline A | Baseline B | Treatment |
+| --- | --- | --- | --- |
+| Spine preservation | Weak. The output shifts from motivating edit boundaries to warning that conversational revision is not enough to prevent drift. | Good. The output keeps the section focused on evidence boundaries and visible edit controls. | Strong. The output is explicitly checked against a spine card before acceptance. |
+| Claim drift | High. The output broadens into claims that AI writing tools can become unsafe and that revised paragraphs should be treated as unreliable unless the workflow can prove preservation. | Low. The output keeps the narrower claim that workflows need explicit controls. | Low and audited. The drift audit records `changed_claims=none`. |
+| Evidence boundary | Weak. Source A and Source B are treated as support for model unreliability entering thesis prose, which is stronger than the stated source boundary. | Good. The source boundary remains visible and rejects universal model-safety claims. | Strong. The source boundary is encoded in `core_claims`, `forbidden_changes`, and the final audit. |
+| Scope discipline | Weak. The output invites the next subsection to describe implementation, but also starts to imply a control workflow solution inside the motivation section. | Good. Product architecture remains outside this section. | Strong. The edit contract forbids product architecture and points adjacent implementation details to the next subsection. |
+| Author control recovery | Weak. The author must infer what drift occurred by rereading the edited prose and the conversation. | Partial. The consolidated prompt helps review, but there is no durable applied contract or audit. | Strong. The author can inspect the spine card, applied contract, edited section, and drift audit. |
+
+## Human Review Decision
+
+| Workflow | Decision | Reason |
+| --- | --- | --- |
+| Baseline A | revise | The prose is fluent, but it broadens the claim and weakens the evidence boundary. |
+| Baseline B | partial_accept | The prose preserves the core argument, but the author still needs to manually verify the output against the original requirements. |
+| Treatment | accept_after_author_check | The control packet makes the edit scope, forbidden changes, and drift decision inspectable. The audit reports no high-risk drift, but the final acceptance remains an author decision. |
+
+## Control Finding
+
+Baseline A demonstrates the lost-in-conversation risk: later constraints are present, but the edited output still broadens the argument. Baseline B shows that consolidation helps, but it remains a prompt technique. Treatment shows the product claim: `/thesis-control` does not merely produce smoother prose; it produces inspectable control artifacts that let the author decide whether the edit should be accepted, revised, or rolled back.

examples/lost-in-conversation-bench/treatment/review_report.md

@@ -12,8 +12,8 @@ The same multi-turn requirements are converted into a spine card, edit contract,
 | Claim drift | Strong. The edit preserves the bounded claim and avoids universal AI-writing unreliability. |
 | Evidence boundary | Strong. The edit keeps Source A and Source B tied to workflow-control need rather than broad model-safety claims. |
 | Scope discipline | Strong. Product architecture remains outside this motivation section. |
-| Author control recovery | Strong. The author can inspect `spine_cards.csv`, `edit_contracts.csv`, and `drift_audits.csv` without reconstructing the full chat history. |
+| Author control recovery | Strong. The author can inspect `spine_cards.csv`, `edit_contracts.csv`, and `drift_audits.csv` without reconstructing the full chat history. The audit reports no high-risk drift, but final acceptance remains an author decision. |
 
 ## Control Finding
 
-The treatment improves author control because the approval decision is tied to explicit artifacts: the spine card, applied edit contract, and drift audit. The prose is not accepted merely because it reads better.
+The treatment improves author control because the approval decision is tied to explicit artifacts: the spine card, applied edit contract, and drift audit. The prose is not accepted merely because it reads better, and the author still owns the final accept, partial accept, revise, or rollback decision.

scripts/check_lost_in_conversation_bench.py

@@ -21,8 +21,11 @@
     "chapters/desensitized_section.md",
     "requirements/multi_turn_requirements.md",
     "requirements/consolidated_prompt.md",
+    "baselines/baseline_a_edited_section.md",
     "baselines/baseline_a_review.md",
+    "baselines/baseline_b_edited_section.md",
     "baselines/baseline_b_review.md",
+    "comparison_report.md",
     "treatment/source_excerpts/lost-conversation-section.md",
     "treatment/edited_section.md",
     "treatment/review_report.md",
@@ -45,6 +48,12 @@
     "## Control Finding",
 ]
 
+REQUIRED_COMPARISON_HEADINGS = [
+    "## Summary",
+    "## Three-Way Metric Comparison",
+    "## Human Review Decision",
+]
+
 
 def read_text(path: Path) -> str:
     return path.read_text(encoding="utf-8")
@@ -99,6 +108,23 @@ def validate_review(path: Path, root: Path, issues: List[dict]) -> None:
             add_issue(issues, "missing-review-metric", rel, f"review does not address {metric}")
 
 
+def validate_comparison_report(root: Path, issues: List[dict]) -> None:
+    path = root / "comparison_report.md"
+    if not path.is_file():
+        return
+    text = read_text(path)
+    rel = str(path.relative_to(root))
+    for heading in REQUIRED_COMPARISON_HEADINGS:
+        if heading not in text:
+            add_issue(issues, "missing-comparison-heading", rel, f"comparison report is missing heading {heading}")
+    for workflow in ["Baseline A", "Baseline B", "Treatment"]:
+        if workflow not in text:
+            add_issue(issues, "missing-comparison-workflow", rel, f"comparison report does not address {workflow}")
+    for metric in REQUIRED_METRICS:
+        if metric not in text:
+            add_issue(issues, "missing-comparison-metric", rel, f"comparison report does not address {metric}")
+
+
 def validate_treatment_packet(root: Path, issues: List[dict]) -> None:
     treatment = root / "treatment"
     spine_path = treatment / "thesis_control" / "spine_cards.csv"
@@ -137,6 +163,7 @@ def validate_bench(root: Path) -> dict:
     validate_review(root / "baselines" / "baseline_a_review.md", root, issues)
     validate_review(root / "baselines" / "baseline_b_review.md", root, issues)
     validate_review(root / "treatment" / "review_report.md", root, issues)
+    validate_comparison_report(root, issues)
     validate_treatment_packet(root, issues)
     return {
         "schema_version": 1,

scripts/test.sh

@@ -1062,8 +1062,11 @@ test_T73() {
     [[ -f "$bench/chapters/desensitized_section.md" ]] || return 1
     [[ -f "$bench/requirements/multi_turn_requirements.md" ]] || return 1
     [[ -f "$bench/requirements/consolidated_prompt.md" ]] || return 1
+    [[ -f "$bench/baselines/baseline_a_edited_section.md" ]] || return 1
     [[ -f "$bench/baselines/baseline_a_review.md" ]] || return 1
+    [[ -f "$bench/baselines/baseline_b_edited_section.md" ]] || return 1
     [[ -f "$bench/baselines/baseline_b_review.md" ]] || return 1
+    [[ -f "$bench/comparison_report.md" ]] || return 1
     [[ -f "$bench/treatment/source_excerpts/lost-conversation-section.md" ]] || return 1
     [[ -f "$bench/treatment/edited_section.md" ]] || return 1
     [[ -f "$bench/treatment/review_report.md" ]] || return 1