Improve large-guideline parsing and WHO extraction handling.

Add chunked parse/merge for large guidelines and harden query normalization so real WHO guideline runs produce much broader, more usable output instead of failing on LLM formatting and vocabulary mismatches.

Made-with: Cursor

Author: myceldigital

CHANGES_LOG.md

@@ -2,6 +2,14 @@
 
 ## 2026-03-25
 
+- **Add**: automatic chunked parsing for large guidelines in `parse.py`. Herald now narrows to recommendation-heavy sections, splits by subsection headings, parses chunks independently, and merges metadata/fields/synonyms/decisions with deterministic ID conflict handling.
+- **Fix**: chunk parsing now prefers the longest real recommendations chapter instead of the table-of-contents hit, and sanitizes common LLM schema-shape errors before validation (for example bool field value arrays and `null` recommendation strings). Rationale: the first real WHO chunked runs exposed TOC mis-detection and predictable malformed field shapes.
+- **Fix**: query normalization now handles canonical enum values with underscores, pluralized guideline vocab (`adult` ↔ `adults`), null enum vocab lists, and composite values like `children_and_adolescents`. Rationale: the WHO chunked parse exposed valid recommendations that were being missed at query time due to vocab-shape mismatches.
+- **Fix**: `parse.py` now preserves `field_synonyms`, extracts JSON from common LLM wrapper text, and raises the default LLM output budget to `20000` tokens. Rationale: the real WHO mhGAP parse exposed dropped synonyms, prose-before-JSON responses, and mid-JSON truncation.
+- **Fix**: `query.py` now handles parsed `patient_fields` with `values: null` / `known_values: null`, and `any_match` now supports scalar membership against expected enum lists. Rationale: the WHO parse produced these shapes and they broke or suppressed valid queries.
+- **Fix**: CLI commands now surface expected runtime/data failures as clean `Error: ...` messages instead of raw Python tracebacks. Added `tests/test_cli.py` to lock in missing-API-key and invalid-JSON behavior.
+- **Fix**: `convert.py` now strips form-feed page-break characters from `markitdown` output before whitespace normalization. This reduces real-PDF ingestion noise from the WHO mhGAP guideline. Also corrected stale install hints from `guideline-as-code[...]` to `herald-cpg[...]`.
+- **Add**: `workflow_state.md` — bounded Planner packet for a repo-wide quality pass using the WHO mhGAP guideline PDF; keeps execution and judging disciplined.
 - **Fix**: `herald --version` crashed with `RuntimeError: 'herald_cli' is not installed`. Root cause: `@click.version_option()` inferred package name from module (`herald_cli`) instead of the installed package (`herald-cpg`). Fixed by passing `package_name="herald-cpg"`.
 
 - **Add**: `tests/test_diff.py` — 21 tests covering `diff_guidelines()` (identical, added, removed, modified nodes/metadata/fields, complex mixed diffs) and `format_markdown()` (all output sections). `diff.py` now at 99% coverage.

PROJECT_TECHNICAL_OVERVIEW.md

@@ -20,9 +20,10 @@ src/herald_cli/
 └── export.py         # Export decision tree to FHIR PlanDefinition (CPG-on-FHIR)
 
 tests/
-├── test_convert.py   # Markdown normalization (4 tests)
-├── test_parse.py     # Schema models + reference validation (8 tests)
-├── test_query.py     # Query engine + NLP patient parser (20 tests)
+├── test_convert.py   # Markdown normalization (5 tests)
+├── test_cli.py       # CLI error handling (3 tests)
+├── test_parse.py     # Schema models + chunk split/merge + reference validation
+├── test_query.py     # Query engine + NLP patient parser + enum normalization
 ├── test_validate.py  # Source text verification (5 tests)
 ├── test_diff.py      # Guideline version diffing (21 tests)
 └── test_export.py    # FHIR PlanDefinition export (40 tests)
@@ -41,12 +42,36 @@ examples/
 - `MultiQueryEngine`: Queries multiple guidelines simultaneously with conflict detection.
 - `parse_patient_description()`: NLP parser that converts natural language ("34F ADHD, comorbid anxiety") into structured patient dicts. Handles age/sex extraction, medical abbreviations, negation detection, vital signs, medication history, and guideline-driven dynamic field extraction via synonyms.
 - `parse_csv_patients()`: Batch processing from CSV.
+- Query extraction now tolerates `values: null` / `known_values: null` in parsed `patient_fields`.
+- Condition matching now treats `any_match` with a scalar patient value plus a list of expected enum values as membership, which makes the engine more resilient to imperfect LLM condition encoding.
+- Enum extraction now matches canonical values with underscores normalized to spaces (for example `bipolar_disorder` ↔ `bipolar disorder`), aligns derived enum values to guideline vocab (`adult` → `adults`), and treats composite values like `children_and_adolescents` as matching child/adolescent patients.
+
+### `cli.py` — User-Facing Command Layer
+
+- Wraps expected runtime/data failures (`RuntimeError`, `ValueError`, `OSError`, JSON decode errors) as `click.ClickException`, so the CLI prints clean `Error: ...` messages instead of Python tracebacks.
+- Manual CLI coverage now includes `--help`, `--version`, `convert`, `parse` missing-key failure path, `query`, `validate`, `diff`, `export`, batch query, audit logging, and multi-guideline query.
 
 ### `parse.py` — LLM Extraction
 
 - Pydantic models: `GuidelineDecisionTree`, `DecisionNode`, `Condition`, `Recommendation`, `Branch`, `PatientField`, `GuidelineMeta`.
 - Supports Anthropic (claude-sonnet-4-20250514 default) and OpenAI (gpt-4o default).
 - `_validate_references()`: Ensures all branch `next_decision` IDs point to valid nodes.
+- Provider dependency/install error strings now reference the correct package extras: `herald-cpg[anthropic]` and `herald-cpg[openai]`.
+- Parsed trees now preserve `field_synonyms`, matching the documented schema and what `query.py` expects.
+- The parser now strips common LLM wrapper text and extracts fenced/embedded JSON payloads before `json.loads()`.
+- Default output budget for LLM parse calls is now `20000` tokens so large real-world guideline parses are less likely to truncate mid-JSON.
+- Large guidelines now automatically switch to chunked parsing:
+  - tries to narrow to a recommendations chapter when one is clearly present
+  - splits by numbered subsection headings (fallback: markdown headings, then paragraph-size chunks)
+  - sends each chunk with shared top-of-document metadata context
+  - merges guideline metadata, patient fields, synonyms, and decisions deterministically
+  - renames conflicting decision IDs safely and validates references after merge
+- Before schema validation, parse output is sanitized to clean up common LLM shape mistakes seen in real runs (for example `bool` fields incorrectly carrying `[true, false]` value arrays, or nullable recommendation string fields arriving as `null`).
+
+### `convert.py` — PDF Conversion
+
+- Uses Microsoft `markitdown`, then post-processes output with `_normalize_markdown()`.
+- Normalization now strips embedded form-feed/page-break characters before collapsing whitespace so real PDF conversions do not leak `\f` artifacts into downstream parse input.
 
 ### `export.py` — FHIR Export
 
@@ -64,10 +89,15 @@ examples/
 ```bash
 pip install -e ".[dev]"        # Install with dev deps (pytest, ruff, pytest-cov)
 ruff check src/ tests/         # Lint
-pytest tests/ -v --tb=short    # Run tests (98 tests)
+pytest tests/ -v --tb=short    # Run tests (101 tests)
 pytest tests/ --cov=herald_cli # Coverage report
 ```
 
+## Workflow Discipline
+
+- `workflow_state.md` tracks the active Planner-owned task packet, success criteria, and verification steps for non-trivial work.
+- Real-system validation matters for this project because the highest-risk paths are CLI execution, PDF conversion fidelity, and LLM-backed parsing behavior against real guideline documents.
+
 ## Schema
 
 Decision trees follow Herald Schema v0.1 (see `SCHEMA.md`). Every recommendation traces back to `source_section`, `source_page`, and `source_text` for auditability.
@@ -83,4 +113,9 @@ Decision trees follow Herald Schema v0.1 (see `SCHEMA.md`). Every recommendation
 ## Known Issues
 
 - Repo state: `Desktop/herald` has source files deleted from HEAD. Full project only in Downloads copies.
-- Coverage: `cli.py` at 0% (no CLI integration tests). `query.py` at 47% — many extractors and batch/multi-engine paths uncovered.
+- `query.py` coverage is still relatively low for the full NLP extraction surface and multi-guideline behavior.
+- Real WHO PDF conversion still includes repeated running headers/page numbers and some OCR-style spacing artifacts such as `E xecutive summar y`, even though form-feed control characters are now stripped.
+- Multi-guideline queries return duplicate recommendations if equivalent guidelines are loaded more than once; no deduplication layer exists yet.
+- After chunked parsing, the WHO mhGAP parse improved from 14 to 19 decision nodes and became materially more queryable (anxiety, ADHD, bipolar mania, and canonical-phrase depression queries returned results), but fidelity remains low at 50% with partial-only matches. The remaining issue is extraction quality, not parser/runtime stability.
+- Real-world large-guideline extraction still depends heavily on the model producing good local condition vocab. For example, some WHO diagnoses/age buckets were emitted in awkward canonical forms (`moderate_to_severe_depression`, `children_and_adolescents`) that required extra query normalization.
+- The fully chunked WHO rerun currently requires more than five minutes for the complete 23-chunk pass. Under the current workflow rules, that means it must be treated as a long-running operation and discussed before retrying.

src/herald_cli/cli.py

@@ -8,6 +8,12 @@
 from rich.console import Console
 
 console = Console()
+_CLI_HANDLED_EXCEPTIONS = (RuntimeError, ValueError, OSError, json.JSONDecodeError)
+
+
+def _raise_click_error(exc: Exception) -> None:
+    """Render expected runtime failures as clean CLI errors."""
+    raise click.ClickException(str(exc)) from exc
 
 
 @click.group()
@@ -30,12 +36,15 @@ def cli():
 def convert(input_file: Path, output: Path | None):
     """Convert a guideline PDF to structured markdown."""
     from herald_cli.convert import convert_pdf
-    if output is None:
-        output = input_file.with_suffix(".md")
-    console.print(f"[bold]Converting[/bold] {input_file.name} → {output.name}")
-    result = convert_pdf(input_file)
-    output.write_text(result, encoding="utf-8")
-    console.print(f"[green]✓[/green] Written {output} ({result.count(chr(10))} lines)")
+    try:
+        if output is None:
+            output = input_file.with_suffix(".md")
+        console.print(f"[bold]Converting[/bold] {input_file.name} → {output.name}")
+        result = convert_pdf(input_file)
+        output.write_text(result, encoding="utf-8")
+        console.print(f"[green]✓[/green] Written {output} ({result.count(chr(10))} lines)")
+    except _CLI_HANDLED_EXCEPTIONS as exc:
+        _raise_click_error(exc)
 
 
 @cli.command()
@@ -46,14 +55,17 @@ def convert(input_file: Path, output: Path | None):
 def parse(input_file: Path, output: Path | None, provider: str, model: str | None):
     """Parse guideline markdown into a decision tree JSON."""
     from herald_cli.parse import parse_guideline
-    if output is None:
-        output = input_file.with_suffix(".json")
-    console.print(f"[bold]Parsing[/bold] {input_file.name} with {provider}")
-    md = input_file.read_text(encoding="utf-8")
-    tree = parse_guideline(md, provider=provider, model=model)
-    output.write_text(json.dumps(tree, indent=2, ensure_ascii=False), encoding="utf-8")
-    n = len(tree.get("decisions", []))
-    console.print(f"[green]✓[/green] Extracted {n} decision nodes → {output}")
+    try:
+        if output is None:
+            output = input_file.with_suffix(".json")
+        console.print(f"[bold]Parsing[/bold] {input_file.name} with {provider}")
+        md = input_file.read_text(encoding="utf-8")
+        tree = parse_guideline(md, provider=provider, model=model)
+        output.write_text(json.dumps(tree, indent=2, ensure_ascii=False), encoding="utf-8")
+        n = len(tree.get("decisions", []))
+        console.print(f"[green]✓[/green] Extracted {n} decision nodes → {output}")
+    except _CLI_HANDLED_EXCEPTIONS as exc:
+        _raise_click_error(exc)
 
 
 @cli.command()
@@ -90,7 +102,10 @@ def query(input_files, ask, fmt, log, batch, output):
         console.print("[red]Error: provide at least one guideline JSON.[/red]")
         raise SystemExit(1)
 
-    guidelines = [json.loads(f.read_text(encoding="utf-8")) for f in input_files]
+    try:
+        guidelines = [json.loads(f.read_text(encoding="utf-8")) for f in input_files]
+    except _CLI_HANDLED_EXCEPTIONS as exc:
+        _raise_click_error(exc)
 
     # --- Batch mode ---
     if batch:
@@ -433,9 +448,12 @@ def _show_fields(engine):
 def validate(tree_file: Path, source: Path):
     """Validate parsed tree against its source markdown."""
     from herald_cli.validate import print_validation_report, validate_tree
-    console.print(f"[bold]Validating[/bold] {tree_file.name} against {source.name}\n")
-    results = validate_tree(tree_file, source)
-    print_validation_report(results, console)
+    try:
+        console.print(f"[bold]Validating[/bold] {tree_file.name} against {source.name}\n")
+        results = validate_tree(tree_file, source)
+        print_validation_report(results, console)
+    except _CLI_HANDLED_EXCEPTIONS as exc:
+        _raise_click_error(exc)
 
 
 @cli.command()
@@ -446,61 +464,63 @@ def validate(tree_file: Path, source: Path):
 def diff(old_file: Path, new_file: Path, fmt: str, output: Path | None):
     """Compare two versions of a parsed guideline."""
     from herald_cli.diff import diff_guidelines, format_markdown
+    try:
+        old = json.loads(old_file.read_text(encoding="utf-8"))
+        new = json.loads(new_file.read_text(encoding="utf-8"))
+        result = diff_guidelines(old, new)
 
-    old = json.loads(old_file.read_text(encoding="utf-8"))
-    new = json.loads(new_file.read_text(encoding="utf-8"))
-    result = diff_guidelines(old, new)
-
-    if fmt == "json":
-        out_text = json.dumps(result, indent=2, ensure_ascii=False)
-        if output:
-            output.write_text(out_text, encoding="utf-8")
-        else:
-            click.echo(out_text)
-        return
-
-    if fmt == "markdown":
-        md = format_markdown(result, old_file.name, new_file.name)
-        if output:
-            output.write_text(md, encoding="utf-8")
-            console.print(f"[green]✓[/green] Written {output}")
-        else:
-            click.echo(md)
-        return
-
-    # Text format
-    s = result["summary"]
-    console.print(
-        f"\n[bold]Guideline diff:[/bold] {old_file.name} → {new_file.name}\n"
-    )
-    console.print(
-        f"  [green]+{s['nodes_added']} added[/green]  "
-        f"[red]-{s['nodes_removed']} removed[/red]  "
-        f"[yellow]~{s['nodes_modified']} modified[/yellow]  "
-        f"[dim]={s['nodes_unchanged']} unchanged[/dim]"
-    )
-    if result["metadata_changes"]:
-        console.print("\n[bold]Metadata changes:[/bold]")
-        for c in result["metadata_changes"]:
-            console.print(f"  {c['field']}: {c['old']} → {c['new']}")
-    if result["added"]:
-        console.print("\n[bold green]Added:[/bold green]")
-        for a in result["added"]:
-            console.print(f"  + {a['id']}: {a['description']}")
-    if result["removed"]:
-        console.print("\n[bold red]Removed:[/bold red]")
-        for r in result["removed"]:
-            console.print(f"  - {r['id']}: {r['description']}")
-    if result["modified"]:
-        console.print("\n[bold yellow]Modified:[/bold yellow]")
-        for m in result["modified"]:
-            console.print(f"  ~ {m['id']}:")
-            for c in m["changes"]:
-                if "old" in c and "new" in c:
-                    console.print(f"    {c['field']}:")
-                    console.print(f"      [red]- {str(c['old'])[:80]}[/red]")
-                    console.print(f"      [green]+ {str(c['new'])[:80]}[/green]")
-    console.print()
+        if fmt == "json":
+            out_text = json.dumps(result, indent=2, ensure_ascii=False)
+            if output:
+                output.write_text(out_text, encoding="utf-8")
+            else:
+                click.echo(out_text)
+            return
+
+        if fmt == "markdown":
+            md = format_markdown(result, old_file.name, new_file.name)
+            if output:
+                output.write_text(md, encoding="utf-8")
+                console.print(f"[green]✓[/green] Written {output}")
+            else:
+                click.echo(md)
+            return
+
+        # Text format
+        s = result["summary"]
+        console.print(
+            f"\n[bold]Guideline diff:[/bold] {old_file.name} → {new_file.name}\n"
+        )
+        console.print(
+            f"  [green]+{s['nodes_added']} added[/green]  "
+            f"[red]-{s['nodes_removed']} removed[/red]  "
+            f"[yellow]~{s['nodes_modified']} modified[/yellow]  "
+            f"[dim]={s['nodes_unchanged']} unchanged[/dim]"
+        )
+        if result["metadata_changes"]:
+            console.print("\n[bold]Metadata changes:[/bold]")
+            for c in result["metadata_changes"]:
+                console.print(f"  {c['field']}: {c['old']} → {c['new']}")
+        if result["added"]:
+            console.print("\n[bold green]Added:[/bold green]")
+            for a in result["added"]:
+                console.print(f"  + {a['id']}: {a['description']}")
+        if result["removed"]:
+            console.print("\n[bold red]Removed:[/bold red]")
+            for r in result["removed"]:
+                console.print(f"  - {r['id']}: {r['description']}")
+        if result["modified"]:
+            console.print("\n[bold yellow]Modified:[/bold yellow]")
+            for m in result["modified"]:
+                console.print(f"  ~ {m['id']}:")
+                for c in m["changes"]:
+                    if "old" in c and "new" in c:
+                        console.print(f"    {c['field']}:")
+                        console.print(f"      [red]- {str(c['old'])[:80]}[/red]")
+                        console.print(f"      [green]+ {str(c['new'])[:80]}[/green]")
+        console.print()
+    except _CLI_HANDLED_EXCEPTIONS as exc:
+        _raise_click_error(exc)
 
 
 @cli.command(name="export")
@@ -510,12 +530,17 @@ def diff(old_file: Path, new_file: Path, fmt: str, output: Path | None):
 def export_cmd(input_file: Path, output: Path | None, fmt: str):
     """Export a parsed guideline to FHIR PlanDefinition."""
     from herald_cli.export import export_fhir
-    data = json.loads(input_file.read_text(encoding="utf-8"))
-    result = export_fhir(data)
-    if output is None:
-        output = input_file.with_suffix(f".{fmt}.json")
-    output.write_text(json.dumps(result, indent=2, ensure_ascii=False), encoding="utf-8")
-    console.print(f"[green]✓[/green] Exported {len(result.get('action', []))} actions → {output}")
+    try:
+        data = json.loads(input_file.read_text(encoding="utf-8"))
+        result = export_fhir(data)
+        if output is None:
+            output = input_file.with_suffix(f".{fmt}.json")
+        output.write_text(json.dumps(result, indent=2, ensure_ascii=False), encoding="utf-8")
+        console.print(
+            f"[green]✓[/green] Exported {len(result.get('action', []))} actions → {output}"
+        )
+    except _CLI_HANDLED_EXCEPTIONS as exc:
+        _raise_click_error(exc)
 
 
 if __name__ == "__main__":

src/herald_cli/convert.py

@@ -27,7 +27,7 @@ def convert_pdf(input_path: Path) -> str:
     except ImportError:
         raise RuntimeError(
             "markitdown is required for PDF conversion. "
-            "Install it with: pip install 'guideline-as-code[all]' "
+            "Install it with: pip install 'herald-cpg[all]' "
             "or: pip install markitdown"
         )
 
@@ -49,10 +49,12 @@ def _normalize_markdown(text: str) -> str:
     """Clean up markitdown output for better parsing.
 
     Fixes common markitdown artifacts:
+    - Form feed/page break characters embedded in the text stream
     - Excessive blank lines
     - Inconsistent heading markers
     - Trailing whitespace
     """
+    text = text.replace("\f", "\n")
     lines = text.split("\n")
     cleaned = []
     prev_blank = False