Skip to content

New Feature: Support sending files #167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mattieruth wants to merge 6 commits into
base: main
Choose a base branch
from
+564 −6
Open

New Feature: Support sending files #167

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5258d54
feat(js): Added support for new RTVI send-file feature
mattieruth Jan 14, 2026
d005865
Add optional public field to FileUrl types to inform server if the ur…
mattieruth Jul 17, 2026
ee3a2b6
- Version gate checks protocol < 2.2 (was < 1.3, which let 2.0/2.1 bo…
vipyne Jul 28, 2026
26eca6e
Merge pull request #242 from pipecat-ai/vp-send-file
mattieruth Jul 28, 2026
be5c624
Refactor sendFile and extract ImageFileBytes type
mattieruth Jul 28, 2026
7277f67
Add tests for new sendFile work
mattieruth Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
184 changes: 182 additions & 2 deletions client-js/client/client.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,17 @@ import {
LLMFunctionCallStartedData,
LLMFunctionCallStoppedData,
MediaState,
MimeTypeMapping,
Participant,
PipecatMetricsData,
RTVI_PROTOCOL_VERSION,
RTVIEvent,
RTVIEvents,
RTVIFile,
RTVIFileFormat,
RTVIMessage,
RTVIMessageType,
SendFileOptions,
SendTextOptions,
setAboutClient,
TranscriptData,
Expand Down Expand Up @@ -1091,7 +1095,9 @@ export class PipecatClient extends RTVIEventEmitter {
public cancelUIJobGroup(jobId: string, reason?: string): void {
const payload: UICancelJobGroupData = { job_id: jobId };
if (reason !== undefined) payload.reason = reason;
this._sendMessage(new RTVIMessage(RTVIMessageType.UI_CANCEL_JOB_GROUP, payload));
this._sendMessage(
new RTVIMessage(RTVIMessageType.UI_CANCEL_JOB_GROUP, payload)
);
}

/**
Expand Down Expand Up @@ -1155,6 +1161,180 @@ export class PipecatClient extends RTVIEventEmitter {
);
}

@transportReady
public async sendFile(

@filipi87 filipi87 Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is a bit hard to follow. It is doing three things at once: normalizing the input, deciding inline vs. upload, and sending.

Would it be worth restructuring it ? Maybe something like:

  public async sendFile(file: RTVIFile | File, content: string, options: SendFileOptions = {}) {
    this.assertBotSupportsSendFile();

    const resolved = file instanceof File
      ? await this.resolveBrowserFile(file)
      : await this.resolveRTVIFile(file);

    await this._sendMessage(
      new RTVIMessage(RTVIMessageType.SEND_FILE, { file: resolved, content, options })
    );
  }

  /** Small files go inline as base64; anything larger is uploaded first. */
  private async resolveBrowserFile(file: File): Promise<RTVIFile> {
    if (this.exceedsMessageLimit(estimateBase64Size(file.size))) {
      return this.uploadFile(file);
    }
    return {
      name: file.name,
      format: file.type,
      source: { type: "bytes", bytes: await readAsBase64(file) },
    };
  }

  /** Only inline byte payloads need resolving — url and id sources are already remote. */
  private async resolveRTVIFile(file: RTVIFile): Promise<RTVIFile> {
    const normalized = { ...file, format: toMimeType(file.format) };
    if (normalized.source.type !== "bytes") return normalized;
    if (!this.exceedsMessageLimit(normalized.source.bytes.length)) return normalized;
    return this.uploadFile(toUploadableFile(normalized));
  }

@mattieruth mattieruth Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i like it.

@mattieruth mattieruth Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

file: RTVIFile | File,
content: string,
options: SendFileOptions = {}
) {
this._assertBotSupportsSendFile();

const rawMime = file instanceof File ? file.type : file.format.toLowerCase();
const mimeType = rawMime in MimeTypeMapping
? MimeTypeMapping[rawMime as RTVIFileFormat]
: rawMime;

const resolved = file instanceof File
? await this._resolveBrowserFile(file, mimeType)
: await this._resolveRTVIFile(file, mimeType);

await this._sendMessage(
new RTVIMessage(RTVIMessageType.SEND_FILE, { file: resolved, content, options })
);
}

private _assertBotSupportsSendFile() {
if (
this._botVersion[0] < 2 ||
(this._botVersion[0] === 2 && this._botVersion[1] < 2)
) {
throw new RTVIErrors.UnsupportedFeatureError(
"sendFile",
"bot",
"requires RTVI protocol 2.2.0+"
);
}
}

private _resolveBrowserFile(file: File, mimeType: string): Promise<RTVIFile> {
// Estimate base64 size (~33% overhead) + message wrapper before reading into memory.
const estimatedEncodedSize = Math.ceil(file.size * 1.37) + 1000;
if (estimatedEncodedSize > this._transport.maxMessageSize) {
return this.uploadFile(file);
}
return new Promise<RTVIFile>((resolve, reject) => {
const reader = new FileReader();
reader.onerror = () =>
reject(new RTVIErrors.RTVIError("Could not read file data"));
reader.onload = async (e) => {
try {
if (!e.target?.result) {
throw new RTVIErrors.RTVIError("Could not read file data");
}
const dataUrl = e.target.result as string;
// FileBytes.bytes carries raw base64; strip the data-URL prefix.
const base64Data = dataUrl.split(",")[1] ?? dataUrl;
resolve({
name: file.name,
format: mimeType,
source: { type: "bytes", bytes: base64Data },
});
} catch (err) {
reject(err);
}
};
reader.readAsDataURL(file);
});
}

private async _resolveRTVIFile(file: RTVIFile, mimeType: string): Promise<RTVIFile> {
const normalized = { ...file, format: mimeType };
if (normalized.source.type !== "bytes") return normalized;

const estimatedSize = normalized.source.bytes.length + 1000;
if (estimatedSize <= this._transport.maxMessageSize) return normalized;

const byteString = atob(
normalized.source.bytes.split(",")[1] || normalized.source.bytes
);
const ab = new ArrayBuffer(byteString.length);
const ia = new Uint8Array(ab);
for (let i = 0; i < byteString.length; i++) {
ia[i] = byteString.charCodeAt(i);
}
const blob = new Blob([ab], { type: mimeType });
const uploadable = new File([blob], file.name || "uploaded_file", { type: mimeType });
return this.uploadFile(uploadable);
}

/**
* Upload a file to a specified endpoint or the default files endpoint.
* @param file - The File to upload
* @param uploadFileParams - Optional APIRequest. If not provided, constructs
* endpoint from startBotParams.endpoint by replacing the path with /files
* @returns Promise resolving to RTVIFile with name, format, and FileUrl source
*/
public async uploadFile(
file: File,
uploadFileParams?: APIRequest
): Promise<RTVIFile> {
let uploadUrl: string;
let headers: Headers | undefined;
let timeout: number | undefined;

if (uploadFileParams) {
const { endpoint } = uploadFileParams;
headers = uploadFileParams.headers;
timeout = uploadFileParams.timeout;

if (endpoint instanceof URL) {
uploadUrl = endpoint.toString();
} else if (typeof endpoint === "string") {
uploadUrl = endpoint;
} else if (
typeof Request !== "undefined" &&
endpoint instanceof Request
) {
uploadUrl = endpoint.url;
} else {
throw new RTVIErrors.RTVIError(
"Unable to determine URL from uploadFileParams.endpoint"
);
}
} else {
// Construct from startBotParams
const startBotParams = this._transport.startBotParams;
if (!startBotParams?.endpoint) {
throw new RTVIErrors.RTVIError(
"No uploadFileParams provided and no startBotParams.endpoint available"
);
}

timeout = startBotParams.timeout;

let baseUrl: URL;
if (startBotParams.endpoint instanceof URL) {
baseUrl = startBotParams.endpoint;
headers = startBotParams.headers;
} else if (typeof startBotParams.endpoint === "string") {
baseUrl = new URL(startBotParams.endpoint);
headers = startBotParams.headers;
} else if (
typeof Request !== "undefined" &&
startBotParams.endpoint instanceof Request
) {
baseUrl = new URL(startBotParams.endpoint.url);
headers = new Headers(startBotParams.endpoint.headers);
} else {
throw new RTVIErrors.RTVIError(
"Unable to determine base URL from startBotParams.endpoint"
);
}

// Change the path to /files
uploadUrl = `${baseUrl.origin}/files`;
}

// Create FormData with the file
const formData = new FormData();
formData.append("file", file);

// Create the Request object
// Note: Don't set Content-Type header - browser sets it automatically with boundary
const request = new Request(uploadUrl, {
method: "POST",
mode: "cors",
body: formData,
headers: headers ? Object.fromEntries(headers.entries()) : undefined,
});

const response = await makeRequest(
{ endpoint: request, timeout },
this._abortController
);
return response as RTVIFile;
}

/**
* Disconnects the bot, but keeps the session alive
*/
Expand All @@ -1174,7 +1354,7 @@ export class PipecatClient extends RTVIEventEmitter {
: [0, 0, 0];
this._botVersion = botVersion;
logger.debug(`[Pipecat Client] Bot is ready. Version: ${data.version}`);
if (botVersion[0] < 2) {
if (this._botVersion[0] < 2) {
logger.warn(
`[Pipecat Client] Bot protocol version ${data.version} is older than this client (${RTVI_PROTOCOL_VERSION}). Compatibility issues may occur.`
);
Expand Down
116 changes: 115 additions & 1 deletion client-js/rtvi/messages.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import {
import type { A11ySnapshot, UIJobGroupEnvelope } from "./ui";

// Protocol 2.0.0 adds server-driven bot-output progress (spoken_progress, segment_id).
export const RTVI_PROTOCOL_VERSION = "2.1.0";
export const RTVI_PROTOCOL_VERSION = "2.2.0";
export const RTVI_MESSAGE_LABEL = "rtvi-ai";

/**
Expand All @@ -27,6 +27,7 @@ export enum RTVIMessageType {
// Client-to-server messages
CLIENT_MESSAGE = "client-message",
SEND_TEXT = "send-text",
SEND_FILE = "send-file",
DTMF = "dtmf",
// UI Worker Protocol (client-to-server)
UI_EVENT = "ui-event",
Expand Down Expand Up @@ -262,6 +263,119 @@ export type SendTextOptions = {
audio_response?: boolean;
};

type Serializable =
| string
| number
| boolean
| null
| Serializable[]
| { [key: number | string]: Serializable };

export type RTVIImageFormat =
| "png"
| "jpg"
| "jpeg"
| "webp"
| "gif"
| "heic"
| "heif";
export type RTVIDocFormat =
| "pdf"
| "csv"
| "txt"
| "md"
| "doc"
| "docx"
| "xls"
| "xlsx"
| "json"
| "html"
| "css"
| "javascript";
export type RTVIMediaFormat =
| "mp3"
| "wav"
| "ogg"
| "aac"
| "mp4"
| "webm"
| "avi";
export type RTVIFileFormat = RTVIImageFormat | RTVIDocFormat | RTVIMediaFormat;

export const MimeTypeMapping: Record<RTVIFileFormat, string> = {
// Images
png: "image/png",
jpg: "image/jpeg",
jpeg: "image/jpeg",
webp: "image/webp",
gif: "image/gif",
heic: "image/heic",
heif: "image/heif",
// Documents
pdf: "application/pdf",
csv: "text/csv",
txt: "text/plain",
md: "text/markdown",
doc: "application/msword",
docx: "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
xls: "application/vnd.ms-excel",
xlsx: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
json: "application/json",
html: "text/html",
css: "text/css",
javascript: "application/javascript",
// Media
mp3: "audio/mpeg",
wav: "audio/wav",
ogg: "audio/ogg",
aac: "audio/aac",
mp4: "video/mp4",
webm: "video/webm",
avi: "video/x-msvideo",
};

export type FileSourceType = "bytes" | "url" | "id";

export type FileBytes = {
type: Extract<FileSourceType, "bytes">;
bytes: string;
};

export type ImageFileBytes = FileBytes & {
width?: number;
height?: number;
};
Comment on lines +339 to +347

@filipi87 filipi87 Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does feels a bit odd having width and height inside FileBytes. I believe this is needed in case we are sending images, right ?

Maybe do something like this:

type ImageFileBytes = FileBytes & {
  width: number;
  height: number;
};

or

type FileBytes = {
  type: "bytes";
  bytes: string;
  metadata?: ImageMetadata;
};

type ImageMetadata = {
  width: number;
  height: number;
};

What do you think ?

@mattieruth mattieruth Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i like it, though, even for images width/height is optional. TBH, maybe they can just go away? I originally had it because I thought it was required in some scenarios. In pipecat, size is required as part of the UserImageRawFrame that gets generated from the message, but as an already encoded image, the value isn't used.

In the RTVI processor, I just default the size to 0x0 if it's not provided. Maybe we remove this as part of the spec and always use 0x0?

@filipi87 filipi87 Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to provide a way to send the size to keep consistency with the server side API, even if this isn't currently needed for Pipecat.

@mattieruth mattieruth Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

export type FileUrl = {
type: Extract<FileSourceType, "url">;
url: string | URL;
public?: boolean;
};
export type FileId = {
type: Extract<FileSourceType, "id">;
id: string;
};

export type RTVIFile = {
name?: string;
// RTVI definition takes the Mime type here, but in client-js, we support
// clients providing shorthands defined above and we map them to Mime types
format: string;
source: FileBytes | FileUrl | FileId;

@filipi87 filipi87 Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In summary, we're basically sending either the file itself (as bytes), a URL to a file, or the ID of a file that was previously uploaded somewhere. Is that correct ?

I don't see FileId being used anywhere else. Are we planning to support it directly ?

@mattieruth mattieruth Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. Right now I think FileId only works with a runner that supports uploading larger files and accessing them from on disc. However, I did think that it should be spec'd and written in a way that could support the various llm's File APIs. All of which either take an id (anthropic & openai) or are built in a way i'm not sure makes sense using from a client anyway (gemini).

@mattieruth mattieruth Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But for now, i'm explicitly not supporting it. The id must start with pipecat:. Now, whether this is the right prefix or approach to this... not sure. I'd love feedback.

@filipi87 filipi87 Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Let's keep it like this for now.

We should probably just document it or create an example in pipecat-examples showing how to create this runner.

};

export type SendFileOptions = {
run_immediately?: boolean;
audio_response?: boolean;
// for things like 'detail' in openAI or 'citations' in Bedrock

@filipi87 filipi87 Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@mattieruth mattieruth Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a TODO to actually support. Right now, these options fall on the floor.

custom_options?: { [key: number | string]: Serializable };
};

export type FileSupport = {
formats: string[];
sources: FileSourceType[];
maxSize: number; // bytes
};
Comment on lines +373 to +377

@filipi87 filipi87 Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we are not using this anywhere. Should we keep it?

@mattieruth mattieruth Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my original spec, I wanted a way for clients to get support requirements from the server. I haven't built it out yet. So I dunno. Should I? Should I comment it and leave as a todo? The general idea was so that UIs could pre-filter out files that won't work and avoid unnecessary hefty uploads.

@filipi87 filipi87 Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I comment it and leave as a todo?

I think so. At least it would make it clear that we are not supporting it yet.


/** Valid DTMF keypad keys. */
export type DTMFButton =
| "0"
Expand Down
Loading
Loading