Skip to content

README.md

Latest commit

 

History

History
116 lines (86 loc) · 5.69 KB

README.md

File metadata and controls

116 lines (86 loc) · 5.69 KB
Raw

qwen-omni-mcp

An MCP server that gives Claude Code and other AI agents video, image, audio, and audio-video understanding via Bailian (DashScope) using the multimodal Qwen3.8-Max and Qwen3.5-Omni models.

Qwen3.8-Max reads video natively — no client-side frame extraction. Qwen3.5-Omni adds native audio understanding (and audio-track awareness for video). Pass a public media URL or a local file path; the model does the rest. The server also ships MCP instructions that teach text-only agents to reach for these tools when they need to view/read media — while telling natively multimodal agents to prefer their own vision.

Highlights

  • Native video understanding — send a video URL or local file, get grounded analysis
  • Image understanding — describe, Q&A, OCR; doubles as the "eyes" for text-only agents whose file reader can't display images
  • Audio understanding — transcribe, summarize, analyze speech/sound (mp3/wav/flac/ogg/m4a/aac)
  • Audio-video understanding — analyze a video's visuals and its sound track together
  • Thinking control — optional per-call thinking_budget on every media tool; omitted = provider default
  • Local file support — pass a local path; files are sent inline as base64 (25MB guardrail)
  • npx-launchable — one line in your MCP client config

Install

No global install needed. Run directly with npx:

npx -y qwen-omni-mcp

For local development:

git clone <this-repo>
cd qwen-omni-mcp
npm install            # also installs husky git hooks
cp .env.example .env   # fill in DASHSCOPE_API_KEY
npm run dev            # run from source via tsx

Configuration

All config is via environment variables (loaded from .env by dotenv):

Variable Required Default Description
DASHSCOPE_API_KEY yes — Bailian API key
QWEN_MODEL no qwen3.8-max Model id for video/image analysis
QWEN_OMNI_MODEL no qwen3.5-omni-plus Omni model id for audio/audio-video
DASHSCOPE_BASE_URL no https://dashscope.aliyuncs.com/compatible-mode/v1 OpenAI-compatible endpoint
QWEN_REQUEST_TIMEOUT no 300 Per-request timeout in seconds

Get a key at https://bailian.console.aliyun.com/cn-beijing?tab=model#/api-key.

The Anthropic-compatible /apps/anthropic endpoint does not support video input, so this server uses the OpenAI-compatible endpoint.

Use with Claude Code

Add to your MCP client config:

{
  "mcpServers": {
    "qwen-omni-mcp": {
      "command": "npx",
      "args": ["-y", "qwen-omni-mcp"],
      "env": {
        "DASHSCOPE_API_KEY": "your-key"
      }
    }
  }
}

For local development without publishing:

{
  "mcpServers": {
    "qwen-omni-mcp": {
      "command": "npx",
      "args": ["tsx", "src/index.ts"],
      "env": { "DASHSCOPE_API_KEY": "your-key" }
    }
  }
}

Tools

Tool Description
analyze_video Analyze a video (URL or local file) with a custom prompt
analyze_image Analyze an image (URL or local file) with a custom prompt
analyze_audio Analyze an audio file (URL or local) with a custom prompt (Omni)
analyze_audio_video Analyze a video's visuals + sound (URL or local) with a prompt (Omni)
check_endpoint_status Show configured endpoint/model (key redacted)

Each media tool accepts a public http/https URL or a local file path. Local files are read and sent inline as base64, with a 25MB guardrail (verified up to a 14MB video / ~18MB body on Qwen3.7-Plus, and an 8.8MB video / ~11.7MB base64 body on Qwen3.5-Omni, both HTTP 200). Files larger than 25MB must be hosted at a public URL instead. Local input is validated by extension + magic-byte signature before encoding, so non-media files are rejected.

Each media tool also accepts an optional thinking_budget (positive integer): the maximum tokens the model may spend thinking before answering. Omit it to use the provider default (thinking on at full budget for Qwen3.8 hybrid-thinking models). Thinking tokens are billed but do not count against max_tokens, which limits the answer itself.

analyze_audio / analyze_audio_video use the omni model (QWEN_OMNI_MODEL, default qwen3.5-omni-plus) and force text-only output. Audio is sent as an input_audio block in the data:;base64,<b64> form with a format field (mp3/wav/flac/ogg/m4a/aac).

Development

npm run typecheck     # strict tsc
npm run lint          # eslint, --max-warnings 0
npm run format:check  # prettier
npm test              # unit + mocked e2e (no API cost)
npm run build         # emit dist/
LIVE=1 npm run test:live   # real API calls (costs tokens)

CI (.github/workflows/ci.yml) runs the same gates on Node 20/22. secrets-scan.yml runs gitleaks. smoke-live.yml (manual / weekly) runs one real image call.

See AGENTS.md for the full set of agent rules (never bypass hooks, never commit secrets, etc.).

License

MIT