Runs the deterministic preparation/validation pipeline and a pre-authored feedback-loop fixture on synthetic data with no LLM and no network, then shows the hard validation gate rejecting a tampered result.
python examples/synthetic-demo/run_demo.pyEverything is fictional: an invented user "Sam Rivera" and 23 hand-written prompts, including a planted fake secret so you can watch redaction happen. No real or private interaction logs are read or produced.
Outputs land in _run/ (git-ignored). If a cloud-sync client (e.g. OneDrive)
holds a lock on a previous run, the demo transparently switches to _run-2/,
_run-3/, ... instead of failing.
extract -> merge -> chunk -> classify -> validate -> aggregate -> score feedback -> tamper test
| Stage | Script | What you see |
|---|---|---|
| generate | generate_fixtures.py |
writes synthetic Claude-format logs + an answer key |
| extract | scripts/corpus_extract.py |
23 human-typed prompts, 4 redactions (the planted secret) |
| merge | scripts/merge_corpora.py |
consolidates source corpora into 00_corpus.jsonl |
| chunk | scripts/chunk_corpus.py |
dedupe + a manifest bound to the corpus SHA-256 |
| classify | classify_worker.py |
offline — labels come from the answer key, so no model is needed |
| validate | scripts/validate_classifications.py |
hard gate: schema, completeness, collisions → PASS |
| aggregate | scripts/aggregate_stats.py |
Stage-4 stats (N=23, B:K, decisions, …) |
| score | scripts/score_predictions.py |
one hit, one deliberate miss, and a 🔴 escalation from synthetic loop files |
| tamper | (in run_demo.py) |
corrupts one classification → the gate refuses with exit 2 |
- The classifications are pre-authored fixtures, not a measured accuracy claim.
On real data this stage is an LLM classification swarm; the module's semantic
quality remains human-reviewed by design (see the κ≈0.24 note in
TODO.md). - Redaction runs before classification, so the classifier only ever sees the
redacted text (
[REDACTED_APIKEY],[REDACTED_EMAIL]). - The prediction playbook, action log, and feedback are also pre-authored synthetic fixtures. Their score demonstrates the measurement path; it does not report product accuracy.
Written to examples/synthetic-demo/_run/ (git-ignored). Inspect after a run:
_run/STUDIE/00_corpus.jsonl— the extracted, redacted corpus_run/STUDIE/_chunks/manifest.json— SHA-256-bound chunk manifest_run/STUDIE/04_statistik.md— the Stage-4 aggregate_run/avatar/— synthetic playbook, action log, and feedback used by the scorer