New Feature: Support sending files by mattieruth · Pull Request #167 · pipecat-ai/pipecat-client-web

mattieruth · 2026-01-12T18:24:03Z

adds new RTVI types
adds new sendFile() method
adds better error handling for sending messages that are too large
Currently, only sending files as bytes is tested and working

fix rebase mangling

…l is publicly accessible

…ts through). - FileReader path strips the data-URL prefix so bytes carries raw base64, uses the normalized MIME type, includes the filename, and properly rejects on read errors (previously an unreadable file hung the promise forever). - sendFile no longer mutates the caller's RTVIFile (spreads a copy). - messages.ts: "hief" → "heif", removed the duplicate "ogg". Rebuilt client-js — your running example picks up the new dist on refresh, and since the local pipecat branch now advertises 2.2.0, local end-to-end still works.

filipi87 · 2026-07-28T13:18:09Z

+export type FileBytes = {
+  type: Extract<FileSourceType, "bytes">;
+  bytes: string;
+  width?: number;
+  height?: number;
+};


It does feels a bit odd having width and height inside FileBytes. I believe this is needed in case we are sending images, right ?

Maybe do something like this:

type ImageFileBytes = FileBytes & { width: number; height: number; };

or

type FileBytes = { type: "bytes"; bytes: string; metadata?: ImageMetadata; }; type ImageMetadata = { width: number; height: number; };

What do you think ?

i like it, though, even for images width/height is optional. TBH, maybe they can just go away? I originally had it because I thought it was required in some scenarios. In pipecat, size is required as part of the UserImageRawFrame that gets generated from the message, but as an already encoded image, the value isn't used.

In the RTVI processor, I just default the size to 0x0 if it's not provided. Maybe we remove this as part of the spec and always use 0x0?

I think it would be nice to provide a way to send the size to keep consistency with the server side API, even if this isn't currently needed for Pipecat.

filipi87 · 2026-07-28T13:19:38Z

+export type SendFileOptions = {
+  run_immediately?: boolean;
+  audio_response?: boolean;
+  // for things like 'detail' in openAI or 'citations' in Bedrock


this is a TODO to actually support. Right now, these options fall on the floor.

filipi87 · 2026-07-28T13:22:06Z

+export type FileSupport = {
+  formats: string[];
+  sources: FileSourceType[];
+  maxSize: number; // bytes
+};


It looks like we are not using this anywhere. Should we keep it?

In my original spec, I wanted a way for clients to get support requirements from the server. I haven't built it out yet. So I dunno. Should I? Should I comment it and leave as a todo? The general idea was so that UIs could pre-filter out files that won't work and avoid unnecessary hefty uploads.

Should I comment it and leave as a todo?

I think so. At least it would make it clear that we are not supporting it yet.

filipi87 · 2026-07-28T13:24:42Z

+  // RTVI definition takes the Mime type here, but in client-js, we support
+  // clients providing shorthands defined above and we map them to Mime types
+  format: string;
+  source: FileBytes | FileUrl | FileId;


In summary, we're basically sending either the file itself (as bytes), a URL to a file, or the ID of a file that was previously uploaded somewhere. Is that correct ?

I don't see FileId being used anywhere else. Are we planning to support it directly ?

Correct. Right now I think FileId only works with a runner that supports uploading larger files and accessing them from on disc. However, I did think that it should be spec'd and written in a way that could support the various llm's File APIs. All of which either take an id (anthropic & openai) or are built in a way i'm not sure makes sense using from a client anyway (gemini).

But for now, i'm explicitly not supporting it. The id must start with pipecat:. Now, whether this is the right prefix or approach to this... not sure. I'd love feedback.

Agreed. Let's keep it like this for now.

We should probably just document it or create an example in pipecat-examples showing how to create this runner.

filipi87 · 2026-07-28T13:32:50Z

+      this._botVersion[0] < 1 ||
+      (this._botVersion[0] === 1 && this._botVersion[1] < 3)


Shouldn't this check be like this to use this feature ?

Suggested change

this._botVersion[0] < 1 ||

(this._botVersion[0] === 1 && this._botVersion[1] < 3)

this._botVersion[0] < 2 || (this._botVersion[0] === 2 && this._botVersion[1] < 2)

filipi87 · 2026-07-28T13:35:48Z

+      if (estimatedEncodedSize > this._transport.maxMessageSize) {
+        uploadFile = file;
+      } else {
+        return new Promise<void>((resolve) => {


Suggestion from Claude:

_The inline-file-read path returns new Promise<void>((resolve) => { ... }) with an async reader.onload handler, and never rejects. Three error paths leave the promise permanently unsettled, so await client.sendFile(...) hangs forever:

the bare throw new RTVIErrors.RTVIError("Could not read file data") when e.target?.result is missing (client.ts:1211) rejects only the handler's own internal promise, which FileReader never awaits;

await sendFileMessage() (client.ts:1223) can reject — _sendMessage throws MessageTooLargeError when messageSizeWithinLimit fails (client.ts:806), which is reachable because the file.size * 1.37 branch decision is only a heuristic;

no reader.onerror is registered, so any FileReader failure never settles the promise either.

Suggested fix: wrap the onload body in try/catch and call reject(err) on failure; add reader.onerror = (err) => reject(err)._

i believe this was resolved when I did the refactor.

filipi87 · 2026-07-28T13:45:46Z

+  | "webp"
+  | "gif"
+  | "heic"
+  | "hief";


Typo: I believe it should be heif.

filipi87 · 2026-07-28T13:46:09Z

+  webp: "image/webp",
+  gif: "image/gif",
+  heic: "image/heic",
+  hief: "image/heif",


Same here: heif

filipi87 · 2026-07-28T14:00:42Z

  }

+  @transportReady
+  public async sendFile(


This function is a bit hard to follow. It is doing three things at once: normalizing the input, deciding inline vs. upload, and sending.

Would it be worth restructuring it ? Maybe something like:

public async sendFile(file: RTVIFile | File, content: string, options: SendFileOptions = {}) { this.assertBotSupportsSendFile(); const resolved = file instanceof File ? await this.resolveBrowserFile(file) : await this.resolveRTVIFile(file); await this._sendMessage( new RTVIMessage(RTVIMessageType.SEND_FILE, { file: resolved, content, options }) ); } /** Small files go inline as base64; anything larger is uploaded first. */ private async resolveBrowserFile(file: File): Promise<RTVIFile> { if (this.exceedsMessageLimit(estimateBase64Size(file.size))) { return this.uploadFile(file); } return { name: file.name, format: file.type, source: { type: "bytes", bytes: await readAsBase64(file) }, }; } /** Only inline byte payloads need resolving — url and id sources are already remote. */ private async resolveRTVIFile(file: RTVIFile): Promise<RTVIFile> { const normalized = { ...file, format: toMimeType(file.format) }; if (normalized.source.type !== "bytes") return normalized; if (!this.exceedsMessageLimit(normalized.source.bytes.length)) return normalized; return this.uploadFile(toUploadableFile(normalized)); }

- Version gate checks protocol < 2.2 (was < 1.3, which let 2.0/2.1 bo…

- sendFile split into _assertBotSupportsSendFile, _resolveBrowserFile, and _resolveRTVIFile helpers so each concern is separate and the public method reads linearly. - FileBytes no longer carries width/height; new ImageFileBytes = FileBytes & { width?, height? } for callers that need image dimensions without changing the wire format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

filipi87

LGTM!

Great to see this API out! 🚀

mattieruth · 2026-07-28T20:29:49Z

Note: I don't think I should merge this until the Pipecat changes are in and released.

mattieruth mentioned this pull request Jan 12, 2026

Improved and introduced error handling for too-large messages pipecat-ai/pipecat-client-web-transports#83

Closed

mattieruth force-pushed the send-file branch from e91015c to ff693e4 Compare February 3, 2026 19:06

mattieruth changed the title ~~Initial commit: Adding support for new API to send files~~ New Feature: Support sending files Feb 4, 2026

mattieruth force-pushed the send-file branch from 3debfc5 to 69a86c6 Compare July 16, 2026 19:28

mattieruth requested review from filipi87, kompfner and markbackman July 16, 2026 19:29

mattieruth marked this pull request as ready for review July 16, 2026 19:29

mattieruth force-pushed the send-file branch from 69a86c6 to 26c7364 Compare July 17, 2026 16:27

mattieruth added 2 commits July 17, 2026 15:32

feat(js): Added support for new RTVI send-file feature

5258d54

fix rebase mangling

Add optional public field to FileUrl types to inform server if the ur…

d005865

…l is publicly accessible

mattieruth force-pushed the send-file branch from 26c7364 to d005865 Compare July 17, 2026 20:50

filipi87 reviewed Jul 28, 2026

View reviewed changes

mattieruth and others added 3 commits July 28, 2026 10:46

Merge pull request #242 from pipecat-ai/vp-send-file

26eca6e

- Version gate checks protocol < 2.2 (was < 1.3, which let 2.0/2.1 bo…

Add tests for new sendFile work

7277f67

filipi87 approved these changes Jul 28, 2026

View reviewed changes

markbackman mentioned this pull request Aug 24, 2026

fix(react): Keep an assistant turn in one message on RTVI 2.0.0 #246

Merged

		this._botVersion[0] < 1 \|\|
		(this._botVersion[0] === 1 && this._botVersion[1] < 3)

	this._botVersion[0] < 1 \|\|
	(this._botVersion[0] === 1 && this._botVersion[1] < 3)
	this._botVersion[0] < 2 \|\| (this._botVersion[0] === 2 && this._botVersion[1] < 2)

Uh oh!

Conversation

mattieruth commented Jan 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

filipi87 Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

filipi87 left a comment

Choose a reason for hiding this comment

Uh oh!

mattieruth commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

filipi87 Jul 28, 2026 •

edited

Loading