Follow-up: strip marker for non-image base64 payloads should record the real mediatype (from PR #719)

[philcunliffe]

Deferred from PR #719 (which fixed #718's 124.5 MB/day regression). Not a blocker for that PR: the marker is truthful about the presence of a stripped payload even when wrong about its type, and no downstream consumer of content_text currently parses or branches on the marker's mime field.

What is wrong

stripBase64DataUris in hypaware-core/plugins-workspace/ai-gateway/src/message_projector.js replaces every data:<mime>;base64,<payload> in content_text with a fixed sentinel:

// message_projector.js:1151-1152
const BASE64_DATA_URI = /data:[^\s,]{0,255}?;base64,[A-Za-z0-9+/=_-]+/g
const STRIPPED_DATA_URI = 'data:image;base64,<stripped>'

The marker always says image, even when the stripped payload was application/pdf, audio/*, or anything else. The row then asserts something false about what was captured: a search for data:application/pdf over content_text silently misses rows where such a payload was present, because it has been rewritten to claim it was an image.

Why it was deferred rather than fixed

The fixed-constant marker is the suggested snippet from #718 itself, and #718 lists "the mime allowlist" as a point worth deciding rather than assuming. Review rounds 1 and 2 on PR #719 both flagged it and left it deliberately; test/plugins/ai-gateway-content-data-uri.test.js pins the current behaviour as intentional.

Reachability today is thin. The only production trigger on record is Codex view_image, which is PNG-only by construction, and nothing in the current codex or claude plugin source emits an input_file / application/pdf / input_audio data URI through this string path. The surrounding JSON ("type":"input_image","image_url":"data:image;base64,<stripped>") is left untouched by the regex, so the structural context a human or the n-gram index relies on for "was there an attachment here" survives intact.

Suggested fix

Verify against current line numbers before landing:

const BASE64_DATA_URI = /data:([^\s,]{0,255}?);base64,[A-Za-z0-9+/=_-]+/g
return text.replace(BASE64_DATA_URI, (_m, mime) => `data:${mime || 'application/octet-stream'};base64,<stripped>`)

This captures the matched mediatype and echoes it back, falling back to application/octet-stream for the empty-mediatype case the existing bare: data:;base64,... test already covers.

The decision needed before landing

Echo the mediatype verbatim (simplest, matches what was on the wire), or apply an allowlist/normalization (collapse image/png, image/jpeg and friends to a known set, bucket everything else as application/octet-stream) to avoid storing an attacker-influenced or malformed mediatype string in content_text. The regex already caps the mediatype at 255 chars, so an allowlist is defense in depth, not a correctness requirement.

Test to update

test/plugins/ai-gateway-content-data-uri.test.js, the case a non-image mime type is stripped as well: its expected value and its "deliberate" comment both need updating to the new intent.

[philcunliffe]

Closing the allowlist reservation as verbatim, permanently (#736).

Re-derived and verified against hypaware-core/plugins-workspace/ai-gateway/src/message_projector.js at the current BASE64_DATA_URI regex, since a prior PR in this series shipped a rationale that turned out false and a review caught it:

• Forgeable in plain prose, allowlist or not. data:application/pdf;base64,<stripped> typed into an ordinary message passes the stripper byte-for-byte untouched: the payload class [A-Za-z0-9+/=_-]+ requires at least one character and excludes <, so <stripped> can never satisfy it and the regex never matches. Confirmed by running the shipped stripBase64DataUris over that literal string: output equals input. So a consumer that trusted a structured-looking marker could already be handed a byte-identical forgery by the wire; normalizing the mediatype through an allowlist would not change that.
• No injection channel to close. The mediatype class [^\s,]{0,255}? excludes all whitespace and ,, so a marker can never carry a newline, tab, or comma. Confirmed by running the stripper over a mediatype containing an embedded \n and one containing an embedded ,: both leave the input untouched rather than producing a spliced or truncated marker.

Both claims hold as stated. Recorded at the regex in message_projector.js and in the ai_gateway_messages query-skill docs (two marker generations, for queriers). Not closing this issue, just leaving the resolution here for the record.

[philcunliffe]

Correction to the closure note above. Its second bullet, "No injection
channel to close ... Both claims hold as stated", overreaches. What was
verified there was narrower than what was claimed.

[^\s,] is a negated class: it excludes exactly what JS \s matches, plus
,. It does not exclude control characters generally, it admits every one
that isn't whitespace. The verification described (an embedded \n and an
embedded ,) only exercised the two characters the class actually excludes,
so it supports the narrow claim and not the universal one written down.

Corrected claim: the mediatype class excludes all whitespace and ,, so a
marker can never carry a newline, carriage return, tab, or comma to splice a
log line or a CSV cell. That much holds and was verified. But the class
admits every other control character, including ESC: data:evil<ESC> [31mX;base64,QUFB strips to data:evil<ESC>[31mX;base64,<stripped>, ESC
intact. An allowlist is exactly the mechanism that would have closed that
channel, so "no injection channel an allowlist would need to close" does not
hold as a flat statement.

Separately, whether that ESC reaches an operator unescaped is a
content_text rendering question (how hyp query sql writes cells to the
terminal), not something this stripper regex controls one way or the other;
this note doesn't take a position on that beyond flagging it as open.

Same correction has been made in the source comment at
message_projector.js (the code construct this note describes). Not closing
this issue.

[philcunliffe]

Two record fixes to the two notes above, both surfaced by round 2 of PR #748.

1. The ESC example renders wrong. In the correction note, the example wrapped
inside its code span, and CommonMark turns a line ending inside a code span into a
space, so GitHub displays data:evil<ESC> [31mX;base64,QUFB. A space is one of
the few characters the mediatype class genuinely does exclude, so the string as
displayed does not match at all and passes through untouched. In a note whose whole
subject is which characters the class admits, the illustration was showing an input
that cannot produce the stated output.

Without the wrap: data:evil<ESC>[31mX;base64,QUFB strips to
data:evil<ESC>[31mX;base64,<stripped>, ESC intact. The source comment in the repo
states it correctly; only this thread's copy was affected.

2. Both notes say "Not closing this issue", and this issue was already closed.
It was auto-closed at 2026-08-13T06:38:38Z by PR #732's merge commit 7b89cd4,
about fourteen hours before either note was posted, and there is no reopened
event. So that sentence describes what those notes did, not the issue's state. It
is closed and staying closed; the notes are here as the resolution record.

Round 1 of PR #748 also cleared the "still open" claim as correct. That was a miss,
corrected here.