docs(examples): MMLU-Pro worked example (Inspect AI task + locked claim + verified sample log)

Adds a complete end-to-end example under `examples/mmlu_pro/`:

- `mmlu_pro_prml.py` — Inspect AI task definition pinned to the `TIGER-Lab/MMLU-Pro` HF dataset (specific revision).
- `claim.prml.yaml` — locked PRML manifest for a 10-sample run (`sha256:218b553d49c6…`).
- `sample_eval_log.json` — compact Inspect-shaped log that satisfies the claim.
- `verify_claim.py` — verifier that re-derives the hash and checks the threshold; exits 0/1.
- `tests/test_examples.py` — CI assertion that the checked-in log passes verification.

This is the worked example the README was implicitly pointing at. Anyone reading the README can now run `python examples/mmlu_pro/verify_claim.py` and watch the chain close.

Co-authored-by: adit24dhaya <adit24dhaya@users.noreply.github.com>

Author: adit24dhaya

examples/mmlu_pro/README.md

@@ -0,0 +1,62 @@
+# MMLU-Pro PRML example
+
+This example shows how to pre-register and verify a small Inspect AI
+MMLU-Pro run with `falsify-inspect`.
+
+## Files
+
+- `mmlu_pro_prml.py` defines an Inspect AI task backed by the
+  `TIGER-Lab/MMLU-Pro` Hugging Face dataset.
+- `claim.prml.yaml` is the locked PRML manifest for a 10-sample run.
+- `sample_eval_log.json` is a compact Inspect-shaped log that matches the
+  manifest.
+- `verify_claim.py` verifies the sample log against the pre-registered hash.
+
+## Run the eval
+
+From a source checkout, install the package with the optional Inspect
+dependency first:
+
+```bash
+python -m pip install -e ".[inspect]"
+```
+
+Then run a small sample:
+
+```bash
+inspect eval examples/mmlu_pro/mmlu_pro_prml.py \
+  --model openai/gpt-4o-mini \
+  --limit 10 \
+  --sample-shuffle 42
+```
+
+Before publishing a result, lock the claim fields that will be verified later:
+
+```bash
+falsify-inspect lock \
+  --metric accuracy \
+  --threshold 0.75 \
+  --threshold-direction ">=" \
+  --dataset TIGER-Lab/MMLU-Pro \
+  --dataset-hash hf:527feea0afed1de15a8c115abf7be4c912123315 \
+  --model-version openai/gpt-4o-mini \
+  --sample-size 10 \
+  --seed 42 \
+  --task mmlu_pro_prml \
+  --output examples/mmlu_pro/claim.prml.yaml
+```
+
+The checked-in manifest hashes to:
+
+```text
+sha256:218b553d49c6f1f4b1f100a1c143995cc4ece4cbda2e8366d0817060783f1026
+```
+
+## Verify the sample claim
+
+```bash
+python examples/mmlu_pro/verify_claim.py
+```
+
+The script exits with `0` when the hash matches and the observed accuracy
+satisfies the pre-registered threshold.

examples/mmlu_pro/claim.prml.yaml

@@ -0,0 +1,11 @@
+dataset: TIGER-Lab/MMLU-Pro
+dataset_hash: hf:527feea0afed1de15a8c115abf7be4c912123315
+inspect_task: mmlu_pro_prml
+metric: accuracy
+model_version: openai/gpt-4o-mini
+pre_registered: '2026-05-16T00:00:00Z'
+prml_version: '0.1'
+sample_size: 10
+seed: 42
+threshold: 0.75
+threshold_direction: '>='

examples/mmlu_pro/mmlu_pro_prml.py

@@ -0,0 +1,49 @@
+"""Inspect AI task for a small MMLU-Pro PRML example.
+
+Run a development sample with:
+
+    inspect eval examples/mmlu_pro/mmlu_pro_prml.py --limit 10 --sample-shuffle 42
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from inspect_ai import Task, task
+from inspect_ai.dataset import Sample, hf_dataset
+from inspect_ai.scorer import choice
+from inspect_ai.solver import multiple_choice
+
+DATASET_PATH = "TIGER-Lab/MMLU-Pro"
+DATASET_REVISION = "527feea0afed1de15a8c115abf7be4c912123315"
+
+
+def record_to_sample(record: dict[str, Any]) -> Sample:
+    """Convert one MMLU-Pro Hugging Face record into an Inspect sample."""
+    return Sample(
+        input=record["question"],
+        choices=record["options"],
+        target=record["answer"],
+        id=record["question_id"],
+        metadata={
+            "subject": record["category"].lower(),
+        },
+    )
+
+
+@task
+def mmlu_pro_prml(shuffle: bool = False) -> Task:
+    """Return a lightweight MMLU-Pro task suitable for PRML examples."""
+    dataset = hf_dataset(
+        path=DATASET_PATH,
+        split="test",
+        sample_fields=record_to_sample,
+        revision=DATASET_REVISION,
+        shuffle=shuffle,
+    )
+
+    return Task(
+        dataset=dataset,
+        solver=[multiple_choice()],
+        scorer=choice(),
+    )

examples/mmlu_pro/sample_eval_log.json

@@ -0,0 +1,26 @@
+{
+  "eval": {
+    "task": "mmlu_pro_prml",
+    "model": "openai/gpt-4o-mini",
+    "dataset": {
+      "name": "TIGER-Lab/MMLU-Pro",
+      "sha": "hf:527feea0afed1de15a8c115abf7be4c912123315"
+    },
+    "config": {
+      "sample_size": 10,
+      "seed": 42
+    }
+  },
+  "results": {
+    "scores": [
+      {
+        "name": "accuracy",
+        "metrics": {
+          "accuracy": {
+            "value": 0.8
+          }
+        }
+      }
+    ]
+  }
+}

examples/mmlu_pro/verify_claim.py

@@ -0,0 +1,37 @@
+"""Verify the checked-in MMLU-Pro example claim."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+from falsify_inspect import verify_eval_log
+
+HERE = Path(__file__).resolve().parent
+EXPECTED_HASH = "sha256:218b553d49c6f1f4b1f100a1c143995cc4ece4cbda2e8366d0817060783f1026"
+
+
+def verify() -> dict[str, Any]:
+    manifest = yaml.safe_load((HERE / "claim.prml.yaml").read_text(encoding="utf-8"))
+    return verify_eval_log(
+        HERE / "sample_eval_log.json",
+        expected_hash=EXPECTED_HASH,
+        threshold=manifest["threshold"],
+        threshold_direction=manifest["threshold_direction"],
+        pre_registered=manifest["pre_registered"],
+        sample_size=manifest["sample_size"],
+        seed=manifest["seed"],
+    )
+
+
+def main() -> int:
+    result = verify()
+    print(json.dumps(result, indent=2, default=str))
+    return 0 if result["ok"] else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tests/test_examples.py

@@ -0,0 +1,22 @@
+"""Tests for checked-in examples."""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+
+def test_mmlu_pro_verification_example_passes():
+    root = Path(__file__).resolve().parents[1]
+    script = root / "examples" / "mmlu_pro" / "verify_claim.py"
+    spec = importlib.util.spec_from_file_location("mmlu_pro_verify_claim", script)
+    assert spec is not None
+    assert spec.loader is not None
+
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+
+    result = module.verify()
+    assert result["hash_match"] is True
+    assert result["threshold_satisfied"] is True
+    assert result["ok"] is True