Skip to content

critique-theater.md

Latest commit

 

History

History
229 lines (178 loc) · 10.5 KB

critique-theater.md

File metadata and controls

229 lines (178 loc) · 10.5 KB
Raw

Design Jury (Critique Theater)

A built-in design panel that reviews every generation before it ships. Five specialists (Designer, Critic, Brand, Accessibility, Copy) score the artifact across each round, and the run keeps iterating until the composite clears the threshold or the orchestrator gives up.

The product label is Design Jury. The internal feature name (code paths, metrics, env vars) stays Critique Theater: apps/daemon/src/critique/, apps/web/src/components/Theater/, SSE channels critique.*, env vars OD_CRITIQUE_*. The user-facing label is sourced from a single i18n key, critiqueTheater.userFacingName, so the product name can be renamed without touching code.

1. What is Design Jury

When an agent emits an <artifact> block, the Critique Theater wraps the artifact in a structured review cycle. Five panelists vote on independent dimensions, the orchestrator composites their scores, and the run either ships (composite ≥ threshold) or runs another round. The user sees:

  • A live panel while the run is in flight: one lane per panelist with the current round's score, the cumulative must-fix count, and a sparkline of per-dim scores.
  • A score ticker showing the composite trajectory across rounds, with a horizontal threshold marker so it is obvious whether the latest round cleared the bar.
  • A collapsed badge once the run settles: "Shipped at round 2, composite 8.6" for the happy path, with status variants for below_threshold, timed_out, and interrupted.

The Theater runs inside the same CLI session the agent already uses (no second process, no second transport). All five panelists are turns in the same conversation, which keeps the model context coherent and prevents the "panelist disagrees with itself across processes" failure mode.

2. How it works

The five panelists

Role What it scores
Designer Layout, composition, hierarchy. The "is this beautiful and balanced" lens.
Critic Whether the artifact actually meets the brief. Contrast, weight, readability.
Brand Token compliance, voice, on-brand color use.
Accessibility WCAG conformance, focus rings, semantic structure, alt text.
Copy Voice, tone, terseness, error message quality.

The cast is fixed in v1. Per-skill cast configuration is reserved for v2 (see roadmap).

Auto-converging rounds

The orchestrator runs up to 3 rounds by default. After each round the composite is computed from the per-panelist weights:

composite = designer × 0.0
          + critic × 0.4
          + brand × 0.2
          + a11y × 0.2
          + copy × 0.2

(Designer is weighted at zero in v1 because their dimensions are aesthetic preferences rather than ship gates. The slot exists so the Designer's qualitative notes still travel into the transcript, and a future config release can bump the weight without changing the schema.)

If the composite meets the threshold (8.0 / 10) the run ships. Otherwise the orchestrator emits the round summary, the agent revises, and the next round begins. After 3 rounds without convergence the run takes the fallbackPolicy path (ship_best by default, falling back to whichever round had the highest composite).

One CLI session, one transport

The orchestrator does not spawn extra processes per panelist. Each panelist is a turn in the same agent session, separated by <PANELIST role="..."> tags in the protocol stream. The parser converts those into panelist_* events that the web reducer consumes via SSE. This keeps the operational contract identical to a normal generation: same auth, same env, same logs.

3. Settings reference

Enable / disable

The feature is gated by a four-tier resolver on the daemon side:

  1. Per-skill od.critique.policy (highest priority). A skill that sets policy: required forces the panel on for every generation that uses it; policy: opt-out forces it off; policy: opt-in lets the panel run only at M2 and above.
  2. Per-project override. The web's setCritiqueTheaterEnabled setter (Phase 15.2) writes the toggle to localStorage for the in-session UI and, when called with a projectId, performs a read-merge-write through the existing PATCH /api/projects/:id settings endpoint: GET the current project, merge critiqueTheaterEnabled into the existing metadata blob, PATCH the merged object so other metadata fields (kind, templateId, linkedDirs, etc.) survive. If the prefetch GET fails the setter skips the PATCH entirely instead of stomping the row. The daemon reads metadata.critiqueTheaterEnabled on the next spawn. The dedicated Design Jury section in SettingsDialog.tsx calls this project-aware setter for the active project; non-project embeds may still call the setter without a projectId for session-local UI state.
  3. OD_CRITIQUE_ENABLED env override. Power-user lane / CI fixtures.
  4. Rollout phase default (lowest priority). M0 / M1 = false, M2 = true for policy: opt-in skills, M3 = true everywhere.

The web hook (useCritiqueTheaterEnabled) reads localStorage and governs whether the Theater UI renders for the active session; the daemon-side resolver makes the actual routing decision per generation. The two layers share the same toggle but answer different questions.

Defaults: disabled during M0 dark-launch and M1 settings-toggle phases. Enabled by default per skill during M2, then globally during M3 after ≥ 90% of production adapters maintain conformance for 14 consecutive days (see docs/agent-adapters.md).

Per-skill override

A skill can opt in or out via od.critique.policy in its SKILL.md frontmatter:

od:
  critique:
    policy: required  # or 'opt-in', 'opt-out'

Skills that publish a deterministic artifact (e.g. od-export-pdf) usually set opt-out; skills that generate net-new design output (magazine-poster, saas-landing) set required.

Score threshold and weights

The threshold (8.0) and per-role weights (designer 0 / critic 0.4 / brand 0.2 / a11y 0.2 / copy 0.2) are read-only in v1. v2 will expose them in the Settings panel.

4. Reading the score badge

After a run settles the Theater collapses to a single chip. The chip displays:

  • composite — the final score on the 0 / 10 scale.
  • round — which round closed the run.
  • status badge — colored token explaining how the run ended:
    • Shipped: composite ≥ threshold (the happy path, accent-tinted).
    • Below threshold: composite < threshold after all rounds (amber-tinted).
    • Timed out: total run exceeded totalTimeoutMs (amber-tinted).
    • Interrupted: user pressed Esc or clicked Interrupt mid-run.
    • Degraded: the panel was offline this run (adapter could not produce a valid critique). See troubleshooting below.

The interrupted chip uses a distinct copy ("Interrupted at round N, best composite X.X") so the user is not told the run shipped when it did not.

5. Replay

Every run writes a structured transcript (.ndjson, optionally .gz). Opening the chip's Replay button mounts a read-only Theater that plays the transcript at the chosen speed:

  • Instant: flushes every event synchronously. Useful when reopening a finished run.
  • Live: paces events at the original cadence (Phase 7+ follow-up; v1 treats this the same as a 0-ms intervalMs).
  • { intervalMs: N }: fixed N-ms delay between events.
  • Paused: holds the cursor in place. Resuming picks up exactly where the pause landed (no re-flush of prior events).

The replay surface is keyboard-scrubbable: J / K jump round-by-round, Esc exits the replay.

6. Troubleshooting

"Panel offline this run"

The orchestrator emits a degraded event when one of these happens:

Reason Cause Remediation
malformed_block The adapter emitted a <CRITIQUE> block the parser rejects. Re-run the conformance harness locally (pnpm --filter @open-design/daemon vitest run tests/critique-conformance.test.ts) to confirm the adapter's transcript shape. The Phase 12 dashboard surfaces this status as a Prometheus series once that PR lands; until then the harness is the authoritative source.
oversize_block The block exceeded parserMaxBlockBytes. Usually a runaway model; retry once or raise the budget.
adapter_unsupported The adapter is marked critique:degraded for the 24h TTL window. Wait for the TTL to elapse. The adapter-degraded registry exposes clearDegraded(adapterId) from apps/daemon/src/critique/adapter-degraded.ts for programmatic resets; a od adapters clear-degraded <id> CLI wrapper is planned in a follow-up.
protocol_version_mismatch The adapter is on an older protocol. Update the adapter or pin protocol negotiation.
missing_artifact The run finished but no <artifact> body landed. Almost always a prompt bug; check the skill template.

"Below threshold after 3 rounds"

The run shipped under the bar with fallbackPolicy: ship_best. Tune the brief (more specific guidance, tighter token constraints), switch to a skill whose DESIGN.md aligns better with the brief, or raise maxRounds.

"Interrupted at round N"

The user pressed Esc or clicked Interrupt. The chip surfaces the best composite seen so far. You can resume by re-issuing the brief.

7. FAQ

Why five panelists, why fixed? The five-role cast is the smallest set that covers the lenses every artifact reviewer uses in practice (composition, intent, brand, accessibility, voice). Locking the set keeps the SSE wire shape, the SQLite schema, and the metrics dashboard stable. Per-skill cast configuration is reserved for v2.

Why is my adapter marked degraded for 24h? The conformance harness runs every adapter prerelease against 10 brief templates. If an adapter drops under the 90% shipped or 95% clean-parse thresholds for two consecutive cycles, it gets marked critique:degraded for 24h. The mark auto-clears on the next clean cycle.

Can I add my own panelist? Not in v1. v2 plans a panelist plug-in contract; see docs/roadmap.md for the timeline.

Related

  • docs/spec.md — protocol v1 wire format.
  • docs/architecture.md — orchestrator + parser layout.
  • docs/skills-protocol.md — od.critique.policy frontmatter contract.
  • docs/agent-adapters.md — conformance contract and adapter responsibilities.
  • docs/roadmap.md — v2 panelist extensions.
  • apps/daemon/src/critique/AGENTS.md — daemon-side module map.
  • apps/web/src/components/Theater/AGENTS.md — web-side module map.