Responses API: audio content blocks are silently dropped on the v1 conversion path

[mxmzb]

Summary

On the Responses API, an audio content block on a HumanMessage is dropped from the outgoing request when the message selects the v1 conversion path (response_metadata.output_version === "v1"). Nothing is thrown and nothing is warned — the request is sent, the model answers, and it never saw the attachment.

Every other block kind on that same path is converted. audio is the only one whose branch is empty.

Versions

• @langchain/openai@1.5.10 (current latest at time of filing)
• @langchain/core@1.2.9
• Node 20, macOS

Also reproduces on @langchain/openai@1.2.0, so this is not a recent regression.

Reproduction

Self-contained; no network and no credentials. The transport is stubbed so the outgoing request body can be read directly.

import { ChatOpenAI } from "@langchain/openai";
import { HumanMessage } from "@langchain/core/messages";

let captured;
const fetchStub = async (_url, init) => {
  captured = JSON.parse(init.body);
  return new Response(
    JSON.stringify({
      id: "resp_1", object: "response", created_at: 0, model: "gpt-4o", status: "completed",
      output: [{ type: "message", id: "msg_1", role: "assistant", status: "completed",
                 content: [{ type: "output_text", text: "ok", annotations: [] }] }],
      usage: { input_tokens: 1, output_tokens: 1, total_tokens: 2 },
    }),
    { status: 200, headers: { "content-type": "application/json" } }
  );
};

const model = new ChatOpenAI({
  model: "gpt-4o",
  apiKey: "unused",
  useResponsesApi: true,
  configuration: { fetch: fetchStub },
});

function messageWith(block, outputVersion) {
  const m = new HumanMessage({
    content: [{ type: "text", text: "what is in this attachment?" }, block],
  });
  if (outputVersion) m.response_metadata = { output_version: outputVersion };
  return m;
}

const audio = { type: "audio", source_type: "base64", data: "SGVsbG8=",
                mime_type: "audio/wav", metadata: { filename: "clip.wav" } };
const doc   = { type: "file",  source_type: "base64", data: "SGVsbG8=",
                mime_type: "application/pdf", metadata: { filename: "doc.pdf" } };

for (const [label, block, ver] of [
  ["audio, output_version v1", audio, "v1"],
  ["audio, default",           audio, undefined],
  ["file,  output_version v1", doc,   "v1"],
  ["file,  default",           doc,   undefined],
]) {
  captured = undefined;
  await model.invoke([messageWith(block, ver)]);
  const kinds = captured.input[0].content.map((p) => p.type);
  console.log(`${label.padEnd(26)} -> ${JSON.stringify(kinds)}`);
}

Actual

audio, output_version v1   -> ["input_text"]                  <-- attachment gone
audio, default             -> ["input_text","input_audio"]
file,  output_version v1   -> ["input_text","input_file"]
file,  default             -> ["input_text","input_file"]

Expected

The first row should produce ["input_text","input_audio"], matching the default path — or, failing that, throw, the way unsupported combinations do elsewhere in this converter.

Cause

In src/converters/responses.ts (compiled: dist/converters/responses.js:839 in 1.5.10, :656 in 1.2.0), inside iterateItems, the block-kind chain has an empty branch for audio:

} else if (block.type === "server_tool_call_result") {
    yield* flushMessage();
    yield convertFunctionCallOutput(block);
} else if (block.type === "audio") {} else if (block.type === "file") {
    const fileItem = resolveFileItem(block);
    if (fileItem) pushMessageContent([fileItem]);
} else if (block.type === "image") {
    const imageItem = resolveImageItem(block);
    if (imageItem) pushMessageContent([imageItem]);
} else if (block.type === "video") {
    const videoItem = resolveFileItem(block);
    if (videoItem) pushMessageContent([videoItem]);
}

file, image and video each resolve a part. audio matches, does nothing, and falls out of the chain — so the block is consumed without being emitted. "audio" appears exactly once in that file, which is this line, so there is no other handler it could be deferring to.

Why the silence matters

The two neighbouring failure modes are both louder than this one. An unsupported combination on the Chat Completions path throws with a message naming the block type, and an unrecognised block is forwarded verbatim and rejected by the API. Here the request is well-formed and accepted; the only symptom is that the model answers as though no attachment were sent. For an audio attachment that reads as a bad model response rather than a client bug, which is a long way from the cause.

Context

Found while routing non-image attachments through an adapter that hands messages to a LangGraph server. The adapter cannot work around it: it emits messages and does not construct the model, so it has no way to know which transport or output version the graph will use.

Happy to open a PR adding the resolveAudioItem branch if that is a welcome shape for the fix.

[xianjianlf2]

I reproduced the v1 conversion-path data loss and prepared a focused fix that converts audio/wav and audio/mp3 blocks to Responses API input_audio, with a regression test covering a legacy HumanMessage audio block. Unsupported MIME types now fail explicitly rather than silently dropping the attachment. I would like to take this issue and open a PR.