Add VLM-judge creative critique tools - #4373
Merged
Merged
Conversation
Close the generate → look → critique → revise loop with five agent tools built entirely on existing vision-capable chat providers — no dedicated scorer models: - critique_image: directional defect list (what/where/fix) against the brief with a pass/revise verdict; forbids "add more detail" advice. - compare_images: pairwise knockout tournament; every match judged twice with presentation order swapped to cancel position bias, plus a tiebreak call on disagreement. - score_image_adherence: decomposes the brief into binary yes/no checks and answers each with the VLM — an explainable adherence score. - record_style_preference / get_style_profile: persist aesthetic preferences via LongTermMemory and return them as a taste-profile block for generation prompts and judge rubrics. Judging tools take provider+model params (pair with find_model) and run through runProviderPrediction's generate_message capability at temperature 0, so asset:// image sources resolve like everywhere else. All five register as built-ins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGdnz8JAsc4p3SyUUY3sAc
georgi
enabled auto-merge (squash)
July 20, 2026 08:07
CodeQL flags Math.random as insecure randomness. The tiebreak order choice isn't security-sensitive, but node:crypto randomInt costs nothing and clears the alert. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGdnz8JAsc4p3SyUUY3sAc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Five new agent tools in
packages/agentsthat close the creative feedback loop — generate → look → critique → revise — built entirely on the vision-capable chat providers NodeTool already ships. No dedicated scorer models (PickScore/HPSv3-style weights) are introduced; all judging is VLM tool calls.critique_image— reviews one image against the brief and returns directional feedback: concrete defects with location and fix, plus a pass/revise verdict. The prompt forbids "add more detail" advice, since vague critique makes iteration loops polish details while freezing a bad composition.compare_images— picks the best of 2–8 candidates via a pairwise knockout tournament. VLM judges rank far better than they score and their verdicts are order-sensitive, so every match is judged twice with the presentation order swapped; disagreements go to a tiebreak call. Returns the winner plus every match verdict (triage, not deletion).score_image_adherence— decomposes the brief into up to 12 binary yes/no checks (or takes explicit ones) and has the VLM answer each, returning per-check answers and the pass fraction — an explainable adherence score instead of one opaque rating.record_style_preference/get_style_profile— persist aesthetic preferences (which variant the user chose, over what, in which brief) throughLongTermMemoryand return them as a profile block to inject into generation prompts and pass astaste_profileto the judging tools.How
Judging tools take
provider+modelparams (pair withfind_modelon a vision-capable chat model) and callProcessingContext.runProviderPredictionwith thegenerate_messagecapability at temperature 0, soasset://image sources resolve to data URIs through the existing media resolver. Bare asset ids are normalized toasset://URIs. Taste tools follow theltm-toolsbinding pattern (constructor-boundLongTermMemoryor per-user registry fallback, degrading gracefully when LTM isn't configured). All five are registered inBUILTIN_TOOL_CLASSESand exported from the package index.Tests
15 new Vitest cases in
packages/agents/tests/creative-critique-tools.test.tscovering param validation, JSON-parse failure paths, order-swap agreement and position-bias tiebreak behavior, knockout rounds with byes, adherence decompose/answer flow, and both taste tools.npm run test --workspace=packages/agentsshows no new failures (71 pre-existing failures on this branch's base are environmental — sqlite-vec native module unavailable under--ignore-scriptsinstall). Packagetsc --noEmitand lint pass; web and electron typecheck pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01SGdnz8JAsc4p3SyUUY3sAc
Generated by Claude Code