feat: add audio + audio-video analysis tools (qwen3.5-omni) - #6
Conversation
Two new MCP tools (analyze_audio, analyze_audio_video) on qwen3.5-omni-plus,
separate from qwen3.7-plus video/image. Reuse the non-streaming analyze path:
live testing shows the doc's 'stream=True mandatory' claim is stale -- omni
non-stream returns 200 for text/audio/video.
Audio uses input_audio{data,format} where data must be data:;base64,<b64>
(raw base64 rejected). Local files validated by ext + magic-byte before
encoding; 25MB guardrail reused (8.8MB / 11.7MB base64 verified on omni).
New QWEN_OMNI_MODEL env, default qwen3.5-omni-plus.
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe MCP server now supports Omni-based audio and audio-video analysis, including configurable models, audio format detection, local-file encoding, new payload shapes, two new tools, documentation, and expanded automated and live tests. ChangesOmni media analysis
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant MCPServer
participant MediaResolver
participant DashScope
Client->>MCPServer: analyze_audio(audio_url, question)
MCPServer->>MediaResolver: resolveAudio(audio_url)
MediaResolver-->>MCPServer: data and format
MCPServer->>DashScope: Omni request with input_audio and text modality
DashScope-->>MCPServer: answer
MCPServer-->>Client: tool result
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
🟡 Other comments (1)
src/media.ts-40-47 (1)
40-47: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winSend
.m4aaudio asformat: "m4a".DashScope supports
m4aas a distinctinput_audio.formatvalue, so.m4afiles should not be mapped toaac; updateAUDIO_FORMAT[".m4a"]and the docs to avoid sending the wrong audio format label.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/media.ts` around lines 40 - 47, Update the AUDIO_FORMAT mapping so the ".m4a" extension resolves to "m4a" instead of "aac", and update the related documentation to use the same format label.
🧹 Nitpick comments (2)
src/bailian.ts (1)
5-16: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAudio format isn't enforced at the type level;
contentBlock's audio branch is dead/duplicate code.
AnalyzeParams.audioFormatis documented as "Required for audio" (Line 12) but typed optional, and bothcontentBlock(Line 54,audioBlock(url, "")) andbuildPayload(Line 70,params.audioFormat ?? "") silently fall back to an empty format string instead of erroring.contentBlock's own audio branch is unreachable in practice —buildPayloadnever calls it forkind: "audio", it branches independently — so the two places duplicate thekind === "audio"check with different (and one wrong) results, cutting against the stated intent to centralize content-block shapes incontentBlock/audioBlock.Model
AnalyzeParamsas a discriminated union soaudioFormatis required whenkind: "audio", and drop the dead audio fallback incontentBlock.♻️ Suggested discriminated-union fix
-export interface AnalyzeParams { - kind: MediaKind; - url: string; - prompt: string; - maxTokens: number; - /** Per-call model override. Falls back to `cfg.model` when omitted. */ - model?: string; - /** Audio format for `kind: "audio"` (e.g. "mp3", "wav"). Required for audio. */ - audioFormat?: string; - /** Output modalities. Omni calls send `["text"]` to force text-only output. */ - modalities?: string[]; -} +interface CommonAnalyzeParams { + url: string; + prompt: string; + maxTokens: number; + /** Per-call model override. Falls back to `cfg.model` when omitted. */ + model?: string; + /** Output modalities. Omni calls send `["text"]` to force text-only output. */ + modalities?: string[]; +} +export type AnalyzeParams = + | (CommonAnalyzeParams & { kind: "audio"; audioFormat: string }) + | (CommonAnalyzeParams & { kind: "video" | "image" });export function contentBlock(kind: MediaKind, url: string): Record<string, unknown> { if (kind === "video") { return { type: "video_url", video_url: { url } }; } - if (kind === "image") { - return { type: "image_url", image_url: { url } }; - } - // audio uses input_audio with {data, format}; handled in buildPayload via audioBlock. - return audioBlock(url, ""); + return { type: "image_url", image_url: { url } }; }Also applies to: 46-55, 67-86
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/bailian.ts` around lines 5 - 16, Model AnalyzeParams as a discriminated union so the audio variant requires audioFormat while non-audio variants do not accept or require it. Update contentBlock and buildPayload to use the union’s narrowing and preserve the centralized audioBlock/content-block construction, removing empty-string fallbacks and the duplicate unreachable audio branch.src/media.ts (1)
141-177: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
toDataUrlandtoAudioDataduplicate the stat/size/read/signature-check pipeline.Both functions repeat the same stat → isFile → size-guardrail → read → post-read size re-check → signature-check sequence, differing only in the extension validator (
mimeFromExtvsaudioFormatFromExt) and the final data-URL prefix. Extracting the shared steps into one helper would prevent the two guardrails (e.g. the TOCTOU re-check comment at Line 168) from drifting apart if one is updated later.Also applies to: 196-230
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/media.ts` around lines 141 - 177, Extract the shared stat/read/size/signature-validation pipeline from toDataUrl and toAudioData into a private helper, preserving both pre-read and post-read MAX_LOCAL_FILE_BYTES checks and the existing file/signature errors. Parameterize only the extension validator or resulting media metadata needed for each caller, then keep toDataUrl and toAudioData responsible for their distinct data-URL prefixes and format handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Other comments:
In `@src/media.ts`:
- Around line 40-47: Update the AUDIO_FORMAT mapping so the ".m4a" extension
resolves to "m4a" instead of "aac", and update the related documentation to use
the same format label.
---
Nitpick comments:
In `@src/bailian.ts`:
- Around line 5-16: Model AnalyzeParams as a discriminated union so the audio
variant requires audioFormat while non-audio variants do not accept or require
it. Update contentBlock and buildPayload to use the union’s narrowing and
preserve the centralized audioBlock/content-block construction, removing
empty-string fallbacks and the duplicate unreachable audio branch.
In `@src/media.ts`:
- Around line 141-177: Extract the shared stat/read/size/signature-validation
pipeline from toDataUrl and toAudioData into a private helper, preserving both
pre-read and post-read MAX_LOCAL_FILE_BYTES checks and the existing
file/signature errors. Parameterize only the extension validator or resulting
media metadata needed for each caller, then keep toDataUrl and toAudioData
responsible for their distinct data-URL prefixes and format handling.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: QUIET
Plan: Pro Plus
Run ID: a69fef9c-432f-4379-83e1-757e247db1e8
📒 Files selected for processing (13)
.env.exampleAGENTS.mdREADME.mdpackage.jsonsrc/bailian.tssrc/config.tssrc/media.tssrc/server.tstest/bailian.test.tstest/config.test.tstest/live.test.tstest/media.test.tstest/tools.test.ts
📜 Review details
🧰 Additional context used
📓 Path-based instructions (10)
**/*.{ts,js,json,md,env}
📄 CodeRabbit inference engine (AGENTS.md)
Never commit, hardcode, or document real secrets, API keys, tokens, or
.envfiles. ReadDASHSCOPE_API_KEYfrom environment variables viasrc/config.ts; use dummy values such assk-testin fixtures.
Files:
package.jsontest/config.test.tstest/media.test.tstest/live.test.tssrc/config.tstest/bailian.test.tstest/tools.test.tsREADME.mdsrc/server.tssrc/media.tsAGENTS.mdsrc/bailian.ts
package.json
📄 CodeRabbit inference engine (AGENTS.md)
package.json: After cloning, runnpm installso thepreparescript installs Husky hooks, and verifycore.hooksPathis.husky.
Do not add a new runtime, language, or heavy dependency without explicit maintainer approval.
Files:
package.json
**/*
📄 CodeRabbit inference engine (AGENTS.md)
Delete files with
trash, neverrm.
Files:
package.jsontest/config.test.tstest/media.test.tstest/live.test.tssrc/config.tstest/bailian.test.tstest/tools.test.tsREADME.mdsrc/server.tssrc/media.tsAGENTS.mdsrc/bailian.ts
test/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
test/**/*.ts:anymay be used sparingly for fixture typing in tests, but production TypeScript must remain strict.
Unit and mocked end-to-end tests must use MSW to mockfetch; they must not make real API calls.
Every new tool or logic branch must have a test, and overall coverage must remain at least 85%.
Files:
test/config.test.tstest/media.test.tstest/live.test.tstest/bailian.test.tstest/tools.test.ts
test/live.test.ts
📄 CodeRabbit inference engine (AGENTS.md)
Live tests run only with
LIVE=1and a realDASHSCOPE_API_KEY; never include them in the defaultnpm testsuite.
Files:
test/live.test.ts
src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
src/**/*.ts: Use strict TypeScript. Do not useany,@ts-ignore, or non-null assertions insrc/; prefer narrow types andunknownwhen parsing external JSON.
Use the Bailian OpenAI-compatible endpoint${DASHSCOPE_BASE_URL}/chat/completions, defaulting tohttps://dashscope.aliyuncs.com/compatible-mode/v1.
Useqwen3.7-plusfor video and image analysis with native video support; do not perform client-side frame extraction.
Useqwen3.5-omni-plus, configurable throughQWEN_OMNI_MODEL, for audio and audio-video analysis, and sendmodalities: ["text"].
Do not switch multimodal tools to the Anthropic-compatible/apps/anthropicendpoint because it does not support video input.
Usedata:;base64,<b64>plusformatforinput_audio.data, not raw base64; preserve non-streaming Omni calls unless the endpoint later requires streaming.
Files:
src/config.tssrc/server.tssrc/media.tssrc/bailian.ts
src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Preserve existing code style and use Prettier and ESLint auto-fix for formatting.
Files:
src/config.tssrc/server.tssrc/media.tssrc/bailian.ts
src/server.ts
📄 CodeRabbit inference engine (AGENTS.md)
src/server.ts: Preserve the names and argument schemas of the five MCP tools:analyze_video,analyze_image,analyze_audio,analyze_audio_video, andcheck_endpoint_status; add new tools instead of renaming existing ones.
check_endpoint_statusmust redact the API key usingredactKey; keep the no-key-leak test passing.
Files:
src/server.ts
src/media.ts
📄 CodeRabbit inference engine (AGENTS.md)
Enforce the 25MB local-file guardrail and validate local media by extension and magic-byte signature before encoding; centralize audio data URL changes in
toAudioData().
Files:
src/media.ts
src/bailian.ts
📄 CodeRabbit inference engine (AGENTS.md)
src/bailian.ts: Keep the DashScope payload builder injectable. Centralize changes tovideo_url,image_url, andinput_audiocontent-block shapes incontentBlockandaudioBlock.
Keep endpoint, model, content-block, and streaming fallbacks centralized insrc/bailian.ts; verify model IDs and endpoint compatibility before changing these assumptions.
Files:
src/bailian.ts
🧠 Learnings (1)
📓 Common learnings
Learnt from: CR
Repo: sommio/qwen-omni-mcp
Timestamp: 2026-07-29T09:45:28.641Z
Learning: If a secret is accidentally staged, unstage it, rotate the key immediately, and notify the maintainer.
Learnt from: CR
Repo: sommio/qwen-omni-mcp
Timestamp: 2026-07-29T09:45:28.641Z
Learning: Never bypass Git hooks with `git commit --no-verify` or `git push --no-verify`; fix hook failures instead.
Learnt from: CR
Repo: sommio/qwen-omni-mcp
Timestamp: 2026-07-29T09:45:28.641Z
Learning: Before pushing, all quality gates must pass: typecheck, lint, format check, tests, and build; CI runs them on Node 20 and 22.
🔇 Additional comments (11)
package.json (1)
3-4: LGTM!AGENTS.md (1)
36-39: LGTM!Also applies to: 40-45, 46-50, 68-72
README.md (1)
3-13: LGTM!Also applies to: 38-44, 84-91, 92-94
.env.example (1)
8-9: LGTM!src/config.ts (1)
3-14: LGTM!Also applies to: 40-48
test/bailian.test.ts (1)
10-10: LGTM!Also applies to: 57-87
test/config.test.ts (1)
19-29: LGTM!Also applies to: 31-40
test/media.test.ts (1)
220-288: LGTM!src/server.ts (1)
66-105: LGTM!Also applies to: 157-199, 214-214
test/tools.test.ts (1)
79-89: LGTM!Also applies to: 144-190, 289-324
test/live.test.ts (1)
71-109: LGTM!
1 participant
What
Adds two MCP tools on qwen3.5-omni-plus, separate from the existing qwen3.7-plus video/image tools:
analyze_audio— analyze an audio file (URL or local), mp3/wav/flac/ogg/m4a/aacanalyze_audio_video— analyze a video's visuals and its sound track (URL or local)Both accept a public URL or local file path, with a custom
questionprompt andmax_tokens— same shape asanalyze_video/analyze_image.Key decisions
analyzepath. The official Qwen-Omni doc claimsstream=Trueis mandatory, but live testing shows non-streaming calls return HTTP 200 + JSON for text/audio/video. No streaming, no SSE parsing, no test churn on the existing path.input_audio.datamust bedata:;base64,<b64>+format. Raw base64 is rejected ("The provided URL does not appear to be valid"). Verified live for mp3/wav.dashscope.aliyuncs.com/compatible-mode/v1works forqwen3.5-omni-plus; no workspace-specific URL or new base-URL env needed.QWEN_OMNI_MODELenv (defaultqwen3.5-omni-plus);check_endpoint_statusnow reportsomni_model.Live verification
LIVE=1 npm run test:live— all 5 pass, including new audio + audio-video against real assets (audio extracted via ffmpeg into/tmp, never into the asset dir):Gates
typecheck·lint·format:check·test(76 passed, 5 live-skipped) ·build— all green.Out of scope
No streaming, no progress notifications, no
system_promptfield, no voice output (modalities:["text"]). Additive only — existing 3 tools andanalyzepath untouched.Fragile assumptions
Updated in
AGENTS.md. Notably: doc's stream-mandatory claim is stale;input_audioraw-base64 form is rejected; flac/ogg/m4a/aac formats are implemented by-spec but not live-tested (single-point fallback insrc/media.ts).