feat: add audio + audio-video analysis tools

Two new MCP tools (analyze_audio, analyze_audio_video) on qwen3.5-omni-plus,
separate from qwen3.7-plus video/image. Reuse the non-streaming analyze path:
live testing shows the doc's 'stream=True mandatory' claim is stale -- omni
non-stream returns 200 for text/audio/video.

Audio uses input_audio{data,format} where data must be data:;base64,<b64>
(raw base64 rejected). Local files validated by ext + magic-byte before
encoding; 25MB guardrail reused (8.8MB / 11.7MB base64 verified on omni).

New QWEN_OMNI_MODEL env, default qwen3.5-omni-plus.

Author: sommio

.env.example

@@ -5,6 +5,9 @@ DASHSCOPE_API_KEY=
 # Model ID. qwen3.7-plus is multimodal (text+image+video) on Bailian.
 QWEN_MODEL=qwen3.7-plus
 
+# Omni model ID for audio + audio-video analysis (native audio understanding).
+QWEN_OMNI_MODEL=qwen3.5-omni-plus
+
 # OpenAI-compatible endpoint (documented multimodal/video support).
 # Anthropic-compatible /apps/anthropic does NOT support video input.
 DASHSCOPE_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1

AGENTS.md

@@ -33,20 +33,21 @@ CI runs the same on Node 20 and 22. Local green ≠ CI green if you skip a step.
 
 - **TypeScript strict.** No `any` in `src/` (allowed sparingly in `test/` for fixture typing). No `@ts-ignore`. No non-null assertions in `src/`.
 - Prefer narrow types and `unknown` over `any` when parsing external JSON (see `src/bailian.ts`).
-- The DashScope payload builder (`buildPayload`) is intentionally injectable — if the `video_url`/`image_url` content block shape changes, change it in one place.
+- The DashScope payload builder (`buildPayload`) is intentionally injectable — if the `video_url`/`image_url`/`input_audio` content block shape changes, change it in one place (`contentBlock` / `audioBlock` in `src/bailian.ts`).
 - Do not add a new runtime, language, or heavy dependency without explicit maintainer approval.
 - Match existing style; let `prettier` and `eslint --fix` handle formatting.
 
 ## Tool surface
 
-The server exposes 3 MCP tools (see `src/server.ts`): `analyze_video`, `analyze_image`, `check_endpoint_status`. Do not silently change a tool's name or argument schema — that breaks MCP clients. Add new tools rather than renaming.
+The server exposes 5 MCP tools (see `src/server.ts`): `analyze_video`, `analyze_image`, `analyze_audio`, `analyze_audio_video`, `check_endpoint_status`. Do not silently change a tool's name or argument schema — that breaks MCP clients. Add new tools rather than renaming.
 
 `check_endpoint_status` must redact the API key (`redactKey`). There is a test asserting no key leaks — keep it passing.
 
 ## Backend
 
 - Endpoint: Bailian (DashScope) OpenAI-compatible mode, `${DASHSCOPE_BASE_URL}/chat/completions` (default `https://dashscope.aliyuncs.com/compatible-mode/v1`).
-- Model: `qwen3.7-plus` (multimodal, native video — **no client-side frame extraction**).
+- Model: `qwen3.7-plus` (multimodal, native video — **no client-side frame extraction**) for `analyze_video`/`analyze_image`.
+- Omni model: `qwen3.5-omni-plus` (native audio + audio-video understanding) for `analyze_audio`/`analyze_audio_video`, configured via `QWEN_OMNI_MODEL`. Omni calls send `modalities: ["text"]` to force text-only output (no voice blob).
 - The Anthropic-compatible `/apps/anthropic` endpoint does NOT support video input. Do not switch to it for multimodal tools.
 - Video frame sampling is server-side (fixed 0.5s/frame on OpenAI-compatible mode). Do not add frame extraction logic.
 
@@ -64,5 +65,8 @@ The server exposes 3 MCP tools (see `src/server.ts`): `analyze_video`, `analyze_
 ## Fragile assumptions (verify before relying on)
 
 1. The OpenAI-compatible endpoint accepts a `video_url` content block for `qwen3.7-plus`. If a live call rejects it, the fallback is the native DashScope `video` content type or switching to `qwen-vl-max-latest`. Change `contentBlock()` in `src/bailian.ts`.
-2. The exact model id string `qwen3.7-plus`. Verify against the Bailian model list if a call returns a model-not-found error.
-3. Local files up to the 25MB guardrail in `src/media.ts` can be sent as base64 data URLs — verified live (14MB video / ~18MB body, HTTP 200 on `qwen3.7-plus` OpenAI-compatible mode). Larger files must be hosted at a public URL. Local input is validated by extension + magic-byte signature before encoding (see `toDataUrl`).
+2. The exact model id strings `qwen3.7-plus` and `qwen3.5-omni-plus`. Verify against the Bailian model list if a call returns a model-not-found error.
+3. Local files up to the 25MB guardrail in `src/media.ts` can be sent as base64 data URLs — verified live (14MB video / ~18MB body, HTTP 200 on `qwen3.7-plus` OpenAI-compatible mode; 8.8MB video / ~11.7MB base64 body, HTTP 200 on `qwen3.5-omni-plus`). Larger files must be hosted at a public URL. Local input is validated by extension + magic-byte signature before encoding (see `toDataUrl` / `toAudioData`).
+4. **Qwen-Omni `stream=True` is NOT mandatory.** The official doc claims all Qwen-Omni requests must set `stream=True`, but live testing shows non-streaming calls succeed (text/audio/video, HTTP 200 + JSON). The omni tools therefore reuse the same non-streaming `analyze` path as `qwen3.7-plus`. If a future endpoint revision starts rejecting non-stream omni calls, add a streaming variant in `src/bailian.ts` and route omni tools through it.
+5. **`input_audio.data` must be `data:;base64,<b64>` + `format`, not raw base64.** Raw base64 is rejected with `"The provided URL does not appear to be valid"`. Verified live for mp3/wav. If other formats (flac/ogg/m4a/aac) are rejected, change `toAudioData()` in `src/media.ts` (e.g. to a full `data:audio/<fmt>;base64,` data URL) — single point of change, no tool-schema impact.
+6. The default `dashscope.aliyuncs.com/compatible-mode/v1` endpoint serves `qwen3.5-omni-plus` (verified live). No workspace-specific MaaS URL is needed. If a future key/region rejects omni, add an optional `QWEN_OMNI_BASE_URL` env and route omni calls through it.

README.md

@@ -1,15 +1,16 @@
 # qwen-omni-mcp
 
-An [MCP](https://modelcontextprotocol.io) server that gives Claude Code and other AI agents **video and image understanding** via [Bailian (DashScope)](https://bailian.console.aliyun.com/) using the multimodal **Qwen3.7-Plus** model.
+An [MCP](https://modelcontextprotocol.io) server that gives Claude Code and other AI agents **video, image, audio, and audio-video understanding** via [Bailian (DashScope)](https://bailian.console.aliyun.com/) using the multimodal **Qwen3.7-Plus** and **Qwen3.5-Omni** models.
 
-Qwen3.7-Plus reads video natively — **no client-side frame extraction**. Pass a public media URL **or a local file path**; the model does the rest.
+Qwen3.7-Plus reads video natively — **no client-side frame extraction**. Qwen3.5-Omni adds native **audio** understanding (and audio-track awareness for video). Pass a public media URL **or a local file path**; the model does the rest.
 
 ## Highlights
 
 - **Native video understanding** — send a video URL or local file, get grounded analysis
 - **Image understanding** — describe, Q&A, OCR
-- **Local file support** — pass a local path; files are sent inline as base64 data URLs (25MB guardrail)
-- **Convenience tools** — summarize, text extraction, frame comparison, Q&A
+- **Audio understanding** — transcribe, summarize, analyze speech/sound (mp3/wav/flac/ogg/m4a/aac)
+- **Audio-video understanding** — analyze a video's visuals **and** its sound track together
+- **Local file support** — pass a local path; files are sent inline as base64 (25MB guardrail)
 - **npx-launchable** — one line in your MCP client config
 
 ## Install
@@ -34,12 +35,13 @@ npm run dev            # run from source via tsx
 
 All config is via environment variables (loaded from `.env` by `dotenv`):
 
-| Variable               | Required | Default                                             | Description                    |
-| ---------------------- | -------- | --------------------------------------------------- | ------------------------------ |
-| `DASHSCOPE_API_KEY`    | yes      | —                                                   | Bailian API key                |
-| `QWEN_MODEL`           | no       | `qwen3.7-plus`                                      | Model id (multimodal)          |
-| `DASHSCOPE_BASE_URL`   | no       | `https://dashscope.aliyuncs.com/compatible-mode/v1` | OpenAI-compatible endpoint     |
-| `QWEN_REQUEST_TIMEOUT` | no       | `300`                                               | Per-request timeout in seconds |
+| Variable               | Required | Default                                             | Description                         |
+| ---------------------- | -------- | --------------------------------------------------- | ----------------------------------- |
+| `DASHSCOPE_API_KEY`    | yes      | —                                                   | Bailian API key                     |
+| `QWEN_MODEL`           | no       | `qwen3.7-plus`                                      | Model id for video/image analysis   |
+| `QWEN_OMNI_MODEL`      | no       | `qwen3.5-omni-plus`                                 | Omni model id for audio/audio-video |
+| `DASHSCOPE_BASE_URL`   | no       | `https://dashscope.aliyuncs.com/compatible-mode/v1` | OpenAI-compatible endpoint          |
+| `QWEN_REQUEST_TIMEOUT` | no       | `300`                                               | Per-request timeout in seconds      |
 
 Get a key at <https://bailian.console.aliyun.com/cn-beijing?tab=model#/api-key>.
 
@@ -79,13 +81,17 @@ For local development without publishing:
 
 ## Tools
 
-| Tool                    | Description                                               |
-| ----------------------- | --------------------------------------------------------- |
-| `analyze_video`         | Analyze a video (URL or local file) with a custom prompt  |
-| `analyze_image`         | Analyze an image (URL or local file) with a custom prompt |
-| `check_endpoint_status` | Show configured endpoint/model (key redacted)             |
+| Tool                    | Description                                                           |
+| ----------------------- | --------------------------------------------------------------------- |
+| `analyze_video`         | Analyze a video (URL or local file) with a custom prompt              |
+| `analyze_image`         | Analyze an image (URL or local file) with a custom prompt             |
+| `analyze_audio`         | Analyze an audio file (URL or local) with a custom prompt (Omni)      |
+| `analyze_audio_video`   | Analyze a video's visuals + sound (URL or local) with a prompt (Omni) |
+| `check_endpoint_status` | Show configured endpoint/model (key redacted)                         |
 
-Each media tool accepts a public `http`/`https` URL **or a local file path**. Local files are read and sent inline as base64 data URLs, with a 25MB guardrail (verified up to a 14MB video / ~18MB body, HTTP 200). Files larger than 25MB must be hosted at a public URL instead. Local input is validated by extension + magic-byte signature before encoding, so non-media files are rejected.
+Each media tool accepts a public `http`/`https` URL **or a local file path**. Local files are read and sent inline as base64, with a 25MB guardrail (verified up to a 14MB video / ~18MB body on Qwen3.7-Plus, and an 8.8MB video / ~11.7MB base64 body on Qwen3.5-Omni, both HTTP 200). Files larger than 25MB must be hosted at a public URL instead. Local input is validated by extension + magic-byte signature before encoding, so non-media files are rejected.
+
+`analyze_audio` / `analyze_audio_video` use the omni model (`QWEN_OMNI_MODEL`, default `qwen3.5-omni-plus`) and force text-only output. Audio is sent as an `input_audio` block in the `data:;base64,<b64>` form with a `format` field (mp3/wav/flac/ogg/m4a/aac).
 
 ## Development
 

package.json

@@ -1,7 +1,7 @@
 {
   "name": "qwen-omni-mcp",
-  "version": "0.2.2",
-  "description": "MCP server for video/image understanding via Bailian (DashScope) Qwen3.7-Plus",
+  "version": "0.3.0",
+  "description": "MCP server for video/image/audio understanding via Bailian (DashScope) Qwen3.7-Plus + Qwen3.5-Omni",
   "license": "MIT",
   "repository": {
     "type": "git",

src/bailian.ts

@@ -1,12 +1,18 @@
 import type { AppConfig } from "./config.js";
 
-export type MediaKind = "video" | "image";
+export type MediaKind = "video" | "image" | "audio";
 
 export interface AnalyzeParams {
   kind: MediaKind;
   url: string;
   prompt: string;
   maxTokens: number;
+  /** Per-call model override. Falls back to `cfg.model` when omitted. */
+  model?: string;
+  /** Audio format for `kind: "audio"` (e.g. "mp3", "wav"). Required for audio. */
+  audioFormat?: string;
+  /** Output modalities. Omni calls send `["text"]` to force text-only output. */
+  modalities?: string[];
 }
 
 export interface AnalyzeResult {
@@ -41,20 +47,42 @@ export function contentBlock(kind: MediaKind, url: string): Record<string, unkno
   if (kind === "video") {
     return { type: "video_url", video_url: { url } };
   }
-  return { type: "image_url", image_url: { url } };
+  if (kind === "image") {
+    return { type: "image_url", image_url: { url } };
+  }
+  // audio uses input_audio with {data, format}; handled in buildPayload via audioBlock.
+  return audioBlock(url, "");
+}
+
+/**
+ * Audio content block for the OpenAI-compatible `input_audio` type. DashScope
+ * requires `data` to be a URL or a `data:;base64,<b64>` data URL (NOT raw
+ * base64 — raw base64 is rejected as "URL does not appear to be valid"),
+ * plus a `format` field carrying the actual codec. Verified live for mp3/wav.
+ */
+export function audioBlock(data: string, format: string): Record<string, unknown> {
+  return { type: "input_audio", input_audio: { data, format } };
 }
 
 export function buildPayload(cfg: AppConfig, params: AnalyzeParams): Record<string, unknown> {
-  return {
-    model: cfg.model,
+  const mediaBlock =
+    params.kind === "audio"
+      ? audioBlock(params.url, params.audioFormat ?? "")
+      : contentBlock(params.kind, params.url);
+  const payload: Record<string, unknown> = {
+    model: params.model ?? cfg.model,
     max_tokens: params.maxTokens,
     messages: [
       {
         role: "user",
-        content: [{ type: "text", text: params.prompt }, contentBlock(params.kind, params.url)],
+        content: [{ type: "text", text: params.prompt }, mediaBlock],
       },
     ],
   };
+  if (params.modalities) {
+    payload.modalities = params.modalities;
+  }
+  return payload;
 }
 
 function chatCompletionsUrl(cfg: AppConfig): string {

src/config.ts

@@ -3,11 +3,15 @@ import "dotenv/config";
 export interface AppConfig {
   apiKey: string;
   model: string;
+  omniModel: string;
   baseUrl: string;
   timeoutMs: number;
 }
 
+/** Multimodal model for video/image analysis (text+image+video, no audio). */
 export const DEFAULT_MODEL = "qwen3.7-plus";
+/** Omni model for audio and audio-video analysis (native audio understanding). */
+export const DEFAULT_OMNI_MODEL = "qwen3.5-omni-plus";
 export const DEFAULT_BASE_URL = "https://dashscope.aliyuncs.com/compatible-mode/v1";
 export const DEFAULT_TIMEOUT_SECONDS = 300;
 
@@ -37,6 +41,7 @@ export function loadConfig(): AppConfig {
   return {
     apiKey: required("DASHSCOPE_API_KEY"),
     model: process.env.QWEN_MODEL?.trim() || DEFAULT_MODEL,
+    omniModel: process.env.QWEN_OMNI_MODEL?.trim() || DEFAULT_OMNI_MODEL,
     baseUrl: process.env.DASHSCOPE_BASE_URL?.trim() || DEFAULT_BASE_URL,
     timeoutMs: positiveInt("QWEN_REQUEST_TIMEOUT", DEFAULT_TIMEOUT_SECONDS) * 1000,
   };