[codex] Support raw image offload in v1 train client

[eligotts]

Design update — inline/offload image storage

This PR now follows the prime-rl multimodal image storage policy:

• offload: current behavior, rewrite base64 data images to file:// run assets and require file-backed image URLs.
• inline: keep data:image/...;base64,... URLs in the message payload and validate them without rewriting.

TrainClient now calls the policy-aware image preparation helper, so prime-rl can be the single source of truth via environment/config propagation.

Validation after latest push: uv run pytest tests/v1/test_train_client_multimodal.py -q passed (5 passed). Commit/push hooks also passed (ruff check, ruff format, generated AGENTS/CLAUDE check, ty).

Design update — dropped the None/cache-only image path

This PR and its companions (prime-rl #2836 / verifiers #1746 / renderers #89) no longer use the "send None for already-cached images" mechanism. Every image carries its raw descriptor ref at every slot (current and prior turns); /inference/v1/generate rematerializes each ref from disk every request.

Why: the None path coupled correctness to deployment (LRU cache present, single replica / DP-affinity, no eviction) and surfaced a miss as a hard vLLM EngineDeadError (qwen3-vl mrope dereferences a None image_grid_thw) that the retry net couldn't catch across the engine→API IPC. Dropping it is deployment-agnostic (a miss is impossible) and non-hacky. vLLM's mm_hash encoder cache still skips the expensive GPU re-encode for free — we only forgo the cheap IPC/CPU-reprocess dedup.

Validated: color-codeword (Qwen3-VL-4B) under DP=2, no affinity / no cache reliance: 0 crashes, 0 data=None, multi-turn accumulation correct, reward ~0.84. Also confirmed under TP.

This repo: with every image carrying its ref, no cache miss can occur — removed the retry subsystem (_generate_with_image_ref_retry, _has_descriptor_only_images, _retryable_mm_error_type, _json_error_type, _RETRYABLE_MM_ERROR_TYPES). Rollouts call renderers.client.generate directly. Obsolete retry tests removed.

---

Original description

Summary

• tighten v1 multimodal graph serialization around strict raw descriptor sidecars
• reject processed multimodal payload keys recursively, including nested pixel_values, image_embeds, and image_features
• update v1 multimodal tests to use strict prime_raw_mm_item envelopes instead of descriptor-only Qwen payloads
• keep raw image offload and retry behavior aligned with the companion Renderers and Prime-RL PRs

Companion PRs

• Prime-RL: Support v1 raw multimodal image offload prime-rl#2836
• Renderers: [codex] Support raw image refs for multimodal rendering renderers#89

Notes

• Draft/WIP: this depends on the renderer generic raw multimodal ref contract in the companion PR.
• v1 multimodal sidecars intentionally carry raw descriptors only, not processed image tensors or image-processor payloads.
• Prime/vLLM materialization happens from raw image refs rather than Verifiers-held processor outputs.

Validation

• Commit hooks: ruff check, ruff format, generated AGENTS/CLAUDE check passed.
• Push hook: ty (ci parity) passed.
• End-to-end hosted-style smoke through Prime-RL with /home/ubuntu/verifiers, /home/ubuntu/renderers, and /home/ubuntu/prime-rl-v1-raw-mm-offload completed inference, env rollouts, train batch creation, trainer step 0, and decoded strict trainer-bound raw image refs.

[!NOTE]

Add raw image offload support to v1 train client

• Adds prepare_images_inplace in verifiers/utils/multimodal.py to recursively traverse message structures and rewrite inline image parts to file:// URIs via renderers.mm_store.offload_image_to_run_assets; raises RuntimeError for unsupported or non-file sources.
• Adds prepare_request_body and prepare_messages hooks to Client; TrainClient overrides both to offload image parts before rendering.
• InterceptionServer.handle_request now calls these hooks on the request body and each user message batch, surfacing failures as structured errors.
• Multimodal graph nodes now carry raw image descriptor sidecars (with raw_image_uri, hashes, and placeholders) and reject processed payloads like pixel_values; PendingTurn.previous_multi_modal_data exposes merged prefix descriptors for bridge reuse.
• Risk: any existing multimodal message with processed pixel tensor payloads will now be rejected at serialization/deserialization boundaries.

^{Macroscope summarized 9bc3cc3.}

Update: ingress hardening (2c2824ae)

• prepare_images_inplace now covers the full renderer part treaty: nested image_url dicts, direct-string image_url, direct-string image parts, and typed pydantic parts. Non-string sources raise with the part shape named; renderer-side raw mode hard-requires file:// (no second offload layer).
• Interception server labels prepare_messages failures as InterceptionError instead of misattributing them to the user simulator.
• New shape-coverage test (skips until the renderers pin ships mm_store; passes against the sibling renderers checkout). Suite: 839 passed with pre-existing test_envs/test_opencode_rlm_env failures reproduced on the base branch.

---

Note

Medium Risk
Changes multimodal training contracts (file:// requirement, rejection of processed sidecars) and interception error paths; incorrect offload or bridge MM alignment could break multi-turn VL rollouts.

Overview
Adds prepare_images_inplace to rewrite inline/base64 image parts to content-addressed file:// run assets (via renderers.mm_store) across wire dicts and typed messages, and requires file:// for training ingress.

Train path: New Client.prepare_request_body / prepare_messages hooks; TrainClient and v0 RendererClient call image prep before render/trace. InterceptionServer runs prep on each request and user-simulator message batch, surfacing failures as InterceptionError.

Multimodal training graph: Sidecars are raw image descriptors (raw_image_uri, hashes, placeholders) — processed tensors (pixel_values, etc.) are rejected. PendingTurn.previous_multi_modal_data and bridge kwargs keep prefix MM aligned; multimodal turns can bridge again (removed _has_multimodal_content guard). Per-node MM attribution now includes placeholders; branch merge concatenates them.

Legacy v0 bridge: Rollout output preserves live cumulative multi_modal_data when mapping to v1 traces.

^{Reviewed by Cursor Bugbot for commit 9bc3cc3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 0b1d73f. Configure here.}

[cursor]

Inline image URLs always rejected

High Severity

The v1 train path always requires every image_url to become a file:// run asset after preparation, even when offload leaves a data:image/...;base64,... URL unchanged. That conflicts with the intended inline multimodal storage mode where base64 image URLs stay in the message payload, so inline training rollouts fail at request preparation instead of validating in place.

Additional Locations (1)

• verifiers/v1/clients/train.py#L208-L216

 

^{Reviewed by Cursor Bugbot for commit 0b1d73f. Configure here.}

[eligotts]

removed inline mode so this is irrelevant

[macroscopeapp]

Approvability

Verdict: Needs human review

4 blocking correctness issues found. This PR introduces new multimodal training capabilities with substantial new code and validation changes. Multiple unresolved review comments flag high-severity issues including silently dropped images, rejected inline URLs, and potential backwards compatibility breaks for existing traces.

No code changes detected at 9bc3cc3. Prior analysis still applies.

^{You can customize Macroscope's approvability policy. Learn more.}

[macroscopeapp]

🟡 Medium v1/graph.py:76

_validate_raw_mm_item now unconditionally rejects processed multimodal payloads containing keys like pixel_values, and deserialize_multi_modal_data runs it on every multi_modal_data field during deserialization. Loading a previously persisted multimodal v1 trace whose sidecars contain pixel_values now raises TypeError instead of round-tripping, breaking backwards compatibility for existing saved rollouts. Consider allowing processed payloads through on the deserialization path (e.g. by skipping the processed-key check in the validator's before path) so old traces can still be loaded.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/graph.py around line 76:

`_validate_raw_mm_item` now unconditionally rejects processed multimodal payloads containing keys like `pixel_values`, and `deserialize_multi_modal_data` runs it on every `multi_modal_data` field during deserialization. Loading a previously persisted multimodal v1 trace whose sidecars contain `pixel_values` now raises `TypeError` instead of round-tripping, breaking backwards compatibility for existing saved rollouts. Consider allowing processed payloads through on the deserialization path (e.g. by skipping the processed-key check in the validator's `before` path) so old traces can still be loaded.

[macroscopeapp]

🟡 Medium utils/multimodal.py:64

prepare_images_inplace skips validation when an image_url part has a missing or None image_url field: lines 65-67 only call _prepare_image_source when source is not None, so the malformed part passes through unchecked. Downstream, ChatDialect.parse_request normalizes it to ImageUrlSource(url=""), forwarding a request with an empty image URL instead of rejecting it. Consider calling _require_file_image_url(value) (or otherwise validating) when source is None so malformed parts are rejected.

Suggested change

	if value.get("type") == "image_url":
	source = value.get("image_url")
	if source is not None:
	_prepare_image_source(source, image_dir=image_dir)
	if value.get("type") == "image_url":
	source = value.get("image_url")
	if source is not None:
	_prepare_image_source(source, image_dir=image_dir)
	else:
	_require_file_image_url(value)

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/utils/multimodal.py around lines 64-67:

`prepare_images_inplace` skips validation when an `image_url` part has a missing or `None` `image_url` field: lines 65-67 only call `_prepare_image_source` when `source is not None`, so the malformed part passes through unchecked. Downstream, `ChatDialect.parse_request` normalizes it to `ImageUrlSource(url="")`, forwarding a request with an empty image URL instead of rejecting it. Consider calling `_require_file_image_url(value)` (or otherwise validating) when `source` is `None` so malformed parts are rejected.

[macroscopeapp]

🟡 Medium utils/multimodal.py:52

_prepare_image_part calls _offload_image_url(url, image_dir) before checking whether url is already a file:// path. When a prompt already contains local file:// image URLs and the installed renderer build lacks offload_image_to_run_assets, _offload_image_url raises RuntimeError even though no offload was needed, causing valid pre-offloaded multimodal prompts to fail. Consider returning early when the source is already a file:// URL before invoking _offload_image_url.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/utils/multimodal.py around line 52:

`_prepare_image_part` calls `_offload_image_url(url, image_dir)` before checking whether `url` is already a `file://` path. When a prompt already contains local `file://` image URLs and the installed renderer build lacks `offload_image_to_run_assets`, `_offload_image_url` raises `RuntimeError` even though no offload was needed, causing valid pre-offloaded multimodal prompts to fail. Consider returning early when the source is already a `file://` URL before invoking `_offload_image_url`.

[macroscopeapp]

🟠 High utils/multimodal.py:27

_part_image_field returns "image" unchanged for parts with type == "image", so _prepare_image_part offloads the source but leaves the part in the non-canonical {"type": "image", "image": ...} shape. Downstream v1 chat parsing only preserves image_url parts, so the image is silently dropped from the traced/training prompt even after prepare_images_inplace runs. Consider normalizing image parts to image_url (or mapping image to the image_url field during offload) so downstream parsers retain them.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/utils/multimodal.py around line 27:

`_part_image_field` returns `"image"` unchanged for parts with `type == "image"`, so `_prepare_image_part` offloads the source but leaves the part in the non-canonical `{"type": "image", "image": ...}` shape. Downstream v1 chat parsing only preserves `image_url` parts, so the image is silently dropped from the traced/training prompt even after `prepare_images_inplace` runs. Consider normalizing `image` parts to `image_url` (or mapping `image` to the `image_url` field during offload) so downstream parsers retain them.