feat(providers): OpenSpeaker (TTS/music/image) + fal GPT Image 2 - #479
Open
adnanalpolink wants to merge 2 commits into
Open
feat(providers): OpenSpeaker (TTS/music/image) + fal GPT Image 2#479adnanalpolink wants to merge 2 commits into
adnanalpolink wants to merge 2 commits into
Conversation
…2 tools
Adds five BaseTool providers plus the OpenSpeaker API reference. On a machine
with these keys configured this takes tts from 0/6 to 1/7 and image_generation
from 0/11 to 5/13; all route through the existing selectors with no selector
changes.
OpenSpeaker (api.ai33.pro) — one key fronts several upstream vendors:
- openspeaker_tts: ElevenLabs / Minimax / Edge / Kokoro / Vbee / Fish Audio
voices and cloned voices through a single v3 endpoint. The v3 API has no
`model` field — the provider-prefixed voice_id IS the model selector, so the
tool validates the prefix and reports DEGRADED when a bare ElevenLabs model
id (e.g. "eleven_multilingual_v2") is configured, which would otherwise fail
at call time with an opaque error.
- openspeaker_music: Suno, simple and custom modes, returns all takes.
- openspeaker_image: Imagen. Fetches the model catalogue once and clamps
aspect_ratio/resolution/quality to what each model actually accepts, since
these vary widely (gemini-3.1-flash-lite-image is 1K-only, krea-* take no
resolution at all). Adjustments are reported rather than applied silently.
fal (fal.run/openai/gpt-image-2):
- fal_gpt_image: synchronous, so there is no task queue to stall behind.
Defaults to quality=low rather than fal's `high` default. Snaps sizes to
fal's constraints — notably 1024x576, the obvious "1K 16:9", is 589,824px
and below fal's 655,360 minimum, so it is remapped to 1280x720.
Shared client (tools/openspeaker_client.py) handles the async task contract.
Three behaviours worth noting, each found by hitting them:
- `server_busy` arrives BOTH as HTTP 503 and as HTTP 200 with
{"success": false} in the body. Treating the second as a task payload makes
the poller spin until timeout on a task that already finished.
- A busy poll endpoint must not abort the wait — the generation is already
running and paid for — so transient poll failures are absorbed until the
overall deadline; only genuine task failure stops the loop.
- Completed image tasks return metadata.result_images[].imageUrl (camelCase),
and previewUrl is a downscaled proxy that must not be mistaken for the final
asset.
Also gitignores .playwright-mcp/ browser scratch output.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… uploads Hardening pass on the new providers. These tools run inside an agent loop that ingests untrusted material (reference videos, fetched web pages), so a path or URL arriving in a tool call is not necessarily one a human chose. Each fix below closes a way that untrusted input could reach the filesystem or a remote endpoint. 1. Reference-image exfiltration (openspeaker_image) `reference_images` reads local files and uploads them to a third-party API, making it the one input here that can leak file CONTENTS. It now resolves through safe_media_path() and must carry a real image suffix. Previously a poisoned scene_plan naming "../../.env" or a private key would have shipped it to the provider. Both the traversal and the wrong-filetype cases are covered, since containment alone still permits any in-tree file. 2. Path traversal on output_path (all four tools) output_path was used verbatim to write files, so a caller-supplied "../../.bashrc" would have written outside the workspace. Paths now resolve BEFORE the containment check, so `../` sequences and symlinks pointing out of the tree are both caught. Allowed roots are the project tree and the system temp dir; OPENMONTAGE_EXTRA_MEDIA_ROOTS widens that deliberately rather than by accident. 3. Unbounded download / fail-open (openspeaker_client, fal_gpt_image) download() streamed to disk with no ceiling, so a malformed or hostile response could fill the disk. Now capped (512MB media, 128MB image) with the partial file removed on abort. 4. Download URL scheme The download URL comes back from the API rather than from the caller, so it is treated as untrusted: https only. Note on scope: the output_path pattern in (2) is a pre-existing convention shared with google_imagen, grok_image and openai_image. This commit hardens only the tools it introduces rather than silently refactoring unrelated providers; applying safe_media_path() repo-wide would be a reasonable follow-up if maintainers want it. Verified: traversal, absolute-path escape, .env exfiltration and non-image upload all rejected; https-only and size cap enforced with partial cleanup; all 23 real production asset paths still accepted and a live TTS render still writes correctly, so containment does not break legitimate use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds five
BaseToolproviders and the OpenSpeaker API reference. On a machine with these keys configured this takesttsfrom 0/6 → 1/7 andimage_generationfrom 0/11 → 5/13. Everything routes through the existing selectors — no selector changes needed, since they auto-discover by capability.Built and exercised end to end while producing a real 63s explainer, so these have generated actual narration, music, and stills rather than just passing a smoke test.
Tools
openspeaker_ttsttsopenspeaker_musicmusic_generationopenspeaker_imageimage_generationfal_gpt_imageimage_generationopenspeaker_clientDesign notes worth reviewing
The v3 TTS endpoint has no
modelfield. The provider-prefixedvoice_idis the model selector. A bare ElevenLabs model id likeeleven_multilingual_v2looks plausible in config and fails at call time with an opaque error, soget_status()returnsDEGRADEDandget_info()explains the problem rather than letting it surface mid-render.Image model capabilities vary a lot —
gemini-3.1-flash-lite-imageis 1K-only,gpt-image-2goes to 4K,krea-*take noresolutionat all.openspeaker_imagefetches the catalogue once (cached per process) and clamps unsupported values instead of failing the generation on an HTTP 400. Every adjustment is returned indata.parameter_adjustmentsrather than applied silently.fal rejects the obvious "1K 16:9". 1024×576 is 589,824px, under fal's 655,360 minimum.
_resolve_sizesnaps to 1280×720 (the smallest compliant 16:9) and reports it.Three API behaviours found the hard way
server_busyarrives both as HTTP 503 and as HTTP 200 with{"success": false}in the body. Treating the second as a task payload makes the poller spin until timeout on a task that has already finished.metadata.result_images[].imageUrl(camelCase).previewUrlsits alongside it and is a downscaled proxy — easy to ship as a final asset by accident, so it is explicitly excluded.Not included
Provider-specific Layer 3 skills under
.agents/skills/. Happy to add them if you'd like the prompting guidance to live alongside the tools.Testing
availablewith keys set,unavailablewithout.BaseToolcontract verified: schemas,install_instructions,fallback_tools,get_status,estimate_cost.gemini-3.1-flash-lite-image,gpt-image-2,krea-2-medium.🤖 Generated with Claude Code