feat: support the TTS provider integration type (B.20) - #21
Conversation
Mirror of GladysAssistant/Gladys#2746: integrations declaring type: "tts" in their manifest receive the external-integration.tts.synthesize command and answer it through the new onTtsSynthesize(cb) handler — (text, language) => Promise<string> resolving the audio data-URI, acked back as data.audio (30 s ack delay). The SDK enforces the core bounds before acking (curated content types audio/mpeg|wav|ogg|aac, decoded audio of 1 byte to 5 MB) so a bad audio fails with an explicit message on the integration side. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LCCieF9vzy56ULFi9r6b2g
📝 WalkthroughWalkthroughChangesTTS synthesis
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant GladysServer
participant GladysIntegration
participant onTtsSynthesize
GladysServer->>GladysIntegration: TTS_SYNTHESIZE request
GladysIntegration->>onTtsSynthesize: text and normalized language
onTtsSynthesize-->>GladysIntegration: audio data URI
GladysIntegration->>GladysIntegration: validate audio format, base64 data, and decoded size
GladysIntegration-->>GladysServer: acknowledgement with audio
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/gladys-integration.js`:
- Around line 1277-1288: Update the Base64 handling in the onTtsSynthesize flow
to validate the payload before calling Buffer.from, rejecting non-empty inputs
containing invalid Base64 characters or malformed padding while preserving valid
decoding and existing size/empty checks. Add a test covering a non-empty
malformed payload such as invalid characters.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3ee24485-ad49-4b70-9c36-aca132a2f004
📒 Files selected for processing (6)
README.mdindex.d.tslib/constants.jslib/gladys-integration.jstest/tts.test.jstest/types/api.test-d.ts
| const separatorIndex = audio.indexOf(';base64,'); | ||
| const contentType = separatorIndex === -1 ? '' : audio.substring(0, separatorIndex); | ||
| if (!TTS_AUDIO_CONTENT_TYPES.includes(contentType)) { | ||
| throw new Error(`onTtsSynthesize: the audio content type must be one of ${TTS_AUDIO_CONTENT_TYPES.join(', ')}`); | ||
| } | ||
| const decoded = Buffer.from(audio.substring(separatorIndex + ';base64,'.length), 'base64'); | ||
| if (decoded.length === 0) { | ||
| throw new Error('onTtsSynthesize: the synthesized audio is empty'); | ||
| } | ||
| if (decoded.length > MAX_TTS_AUDIO_SIZE) { | ||
| throw new Error(`onTtsSynthesize: maximum decoded audio size is ${MAX_TTS_AUDIO_SIZE} bytes (5 MB)`); | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
According to the Node.js Buffer documentation, does Buffer.from(value, 'base64') reject invalid Base64 characters and invalid padding, or can it decode permissively?
💡 Result:
According to the official Node.js Buffer documentation, the Buffer.from(value, 'base64') method assumes that the provided input string is valid [1][2]. It does not perform strict validation to reject invalid Base64 characters or incorrect padding; instead, it operates permissively [1][2]. Key characteristics of this behavior include: 1. Permissive Decoding: The implementation is designed to be lenient. For example, it automatically ignores whitespace characters—such as spaces, tabs, and new lines—contained within the Base64-encoded string [1][2]. 2. Input Assumptions: Because the method assumes valid input for performance reasons, providing malformed data may result in unpredictable output rather than a thrown error [1][2]. 3. Non-Base64 Data: If a string contains characters that are neither valid Base64 nor ignorable whitespace, the resulting Buffer may contain data that does not correspond to the original intent, and the length of the created Buffer may differ from what one might expect if the input were strictly validated [1][2]. 4. Padding: The decoder is generally lenient regarding padding characters (the '=' character), accepting both padded and unpadded Base64 strings [3]. In summary, the Node.js Buffer API does not strictly validate Base64 input and will decode it permissively, meaning it does not automatically reject invalid characters or improperly padded strings [1][3][2].
Citations:
- 1: https://nodejs.org/api/buffer.html
- 2: https://github.com/nodejs/node/blob/main/doc/api/buffer.md
- 3: https://deverrors.com/errors/nodejs-buffer-base64-invalid-padding
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the relevant constants and validator block without running repository code.
if [ -f lib/gladys-integration.js ]; then
sed -n '1,35p;1036,1058p;1268,1292p' lib/gladys-integration.js | cat -n
else
fd -a 'gladys-integration\.js$' . | head -5
fi
# Probe Node.js Buffer.from base64 behavior for the malformed example(s) from the comment.
node - <<'JS'
const { Buffer } = require('buffer');
const inputs = [
'Zm9v!',
'Zm9v!!',
'Zm9v!!!',
'Zm9v!!!!',
'Zm9v----!!!!',
' Zm9v ',
'Zm9v',
'Zm9v==',
'Zm9v= =',
'',
];
for (const input of inputs) {
const b = Buffer.from(input, 'base64');
console.log(JSON.stringify({
input,
decodedLength: b.length,
decodedHex: b.toString('hex'),
decodedString: b.toString('utf8'),
strictBase64Like: /^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3})?$/.test(input),
}));
}
JSRepository: GladysAssistant/integration-sdk-js
Length of output: 5133
Validate the Base64 payload before decoding.
Buffer.from(..., 'base64') accepts invalid characters and whitespace, so malformed audio/mpeg;base64,Zm9v! passes validation and returns bytes. Reject malformed Base64 before decoding, and add a test for non-empty malformed input.
Proposed fix
- const decoded = Buffer.from(audio.substring(separatorIndex + ';base64,'.length), 'base64');
+ const base64 = audio.substring(separatorIndex + ';base64,'.length);
+ if (!/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3})?$/.test(base64)) {
+ throw new Error('onTtsSynthesize: the synthesized audio must contain valid base64 data');
+ }
+ const decoded = Buffer.from(base64, 'base64');📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const separatorIndex = audio.indexOf(';base64,'); | |
| const contentType = separatorIndex === -1 ? '' : audio.substring(0, separatorIndex); | |
| if (!TTS_AUDIO_CONTENT_TYPES.includes(contentType)) { | |
| throw new Error(`onTtsSynthesize: the audio content type must be one of ${TTS_AUDIO_CONTENT_TYPES.join(', ')}`); | |
| } | |
| const decoded = Buffer.from(audio.substring(separatorIndex + ';base64,'.length), 'base64'); | |
| if (decoded.length === 0) { | |
| throw new Error('onTtsSynthesize: the synthesized audio is empty'); | |
| } | |
| if (decoded.length > MAX_TTS_AUDIO_SIZE) { | |
| throw new Error(`onTtsSynthesize: maximum decoded audio size is ${MAX_TTS_AUDIO_SIZE} bytes (5 MB)`); | |
| } | |
| const separatorIndex = audio.indexOf(';base64,'); | |
| const contentType = separatorIndex === -1 ? '' : audio.substring(0, separatorIndex); | |
| if (!TTS_AUDIO_CONTENT_TYPES.includes(contentType)) { | |
| throw new Error(`onTtsSynthesize: the audio content type must be one of ${TTS_AUDIO_CONTENT_TYPES.join(', ')}`); | |
| } | |
| const base64 = audio.substring(separatorIndex + ';base64,'.length); | |
| if (!/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3})?$/.test(base64)) { | |
| throw new Error('onTtsSynthesize: the synthesized audio must contain valid base64 data'); | |
| } | |
| const decoded = Buffer.from(base64, 'base64'); | |
| if (decoded.length === 0) { | |
| throw new Error('onTtsSynthesize: the synthesized audio is empty'); | |
| } | |
| if (decoded.length > MAX_TTS_AUDIO_SIZE) { | |
| throw new Error(`onTtsSynthesize: maximum decoded audio size is ${MAX_TTS_AUDIO_SIZE} bytes (5 MB)`); | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/gladys-integration.js` around lines 1277 - 1288, Update the Base64
handling in the onTtsSynthesize flow to validate the payload before calling
Buffer.from, rejecting non-empty inputs containing invalid Base64 characters or
malformed padding while preserving valid decoding and existing size/empty
checks. Add a test covering a non-empty malformed payload such as invalid
characters.
2 participants
SDK counterpart of GladysAssistant/Gladys#2746 (B.20 "TTS provider" type): integrations declaring
type: "tts"in their manifest become selectable as the voice of the instance, and answer synthesis requests over the existing WebSocket command channel.What's added
WEBSOCKET_MESSAGE_TYPES.EXTERNAL_INTEGRATION.TTS_SYNTHESIZE(external-integration.tts.synthesize), routed in_handleMessagelike the other auto-acked commands.gladys.onTtsSynthesize(cb)—(text, language) => Promise<string>resolving the audio data-URI (<content_type>;base64,...), acked back asdata.audio(contract C.8).languageis normalized tonullwhen absent (best-effort hint: the user's language for the voice assistant,nullin scenes)._mapTtsAudio, same pattern as the sync-webhook response mapping): curated content types (audio/mpeg,audio/wav,audio/ogg,audio/aac), decoded audio of 1 byte to 5 MB — the same bounds the core enforces inexternalIntegration.registerProxyService.js, so a bad audio fails the ack with an explicit message on the integration side instead of an opaqueEXTERNAL_INTEGRATION_INVALID_TTS_AUDIOon the Gladys side.onTtsSynthesize,TTS_SYNTHESIZE) + type-test coverage.test/tts.test.js, 10 cases): success ack withdata.audio,languagedefaulting tonull, handler throw, missing handler ("not implemented"), non-string / missing separator / non-curated content type / empty audio / > 5 MB rejections, exactly-5 MB accepted.Checks
npm test: 190/190 passnpm run lint,npm run check-types,npm run prettier-check: clean🤖 Generated with Claude Code
https://claude.ai/code/session_01LCCieF9vzy56ULFi9r6b2g
Generated by Claude Code
Summary by CodeRabbit
New Features
Documentation