Fix Gemini TTS, and make the playground play the answer and remember its config

[hiyouga]

The playground could not run gemini-3.1-flash-tts-preview: the second message of a
session always failed, panel state left over from another model failed the first one, and
a single answer rendered as dozens of 40 ms audio players. Two playground papercuts asked
for in review are fixed along with it: a reload no longer wipes the config panel, and a
finished clip plays itself.

What the live API actually allows

Probed against gemini-3.1-flash-tts-preview on 2026-08-20; each of these came back
400 INVALID_ARGUMENT:

Request carries	Message
more than one turn	Multiturn chat is not enabled for this model
an audio part	Audio input modality is not enabled for this model
system_instruction	Developer instruction is not enabled for this model
thinking_config.thinking_level	Thinking level is not supported for this model.
thinking_config.include_thoughts	Thinking is not enabled for this model
tools	Function calling is not enabled for this model

max_output_tokens, response_modalities, and speech_config were accepted. The docs say
only "TTS models can only receive text inputs"; the multi-turn and developer-instruction
limits are not in them, which is why the playground hit them and the client did not expect
them.

What changed

• Gemini3_7Client synthesizes the newest message and drops the rest of the conversation
  for a TTS model. The stateful helpers record the answer's audio, and replaying that audio
  is what broke the next turn.
• A TTS request is built from the speech settings and max_tokens alone, returning early;
  temperature, fast_mode, and prompt_caching validation moves above that branch so
  every model still gets it.
• concat_uni_events_to_uni_message / concatUniEventsToUniMessage merge consecutive
  inline_data items sharing an audio/ mime type, so one utterance is one history item
  and one tracer entry.
• The playground collects a response's audio chunks behind a 🔊 Receiving audio... 2.3s
  line and renders one player when the stream ends; an interrupted stream keeps what had
  arrived. Chunks are joined as bytes before the WAV header is written, and the sample rate
  and channel count are read from the mime type parameters instead of a hardcoded 24 kHz.

Also in this change:

• The finished clip plays once by itself. A stream stopped from the Stop button stays quiet,
  and a browser that blocks autoplay leaves the player ready to press. The token footer is
  appended to the message card instead of re-parsed into it, which would have torn out the
  playing element.
• The configuration panel is written to localStorage under agenthub.playground.config on
  every edit and restored on load, so a reload keeps the model, client type, API key, base
  URL, extra headers, thinking and tool settings, system prompt, tools, and trace id. Text
  fields are stored as typed, so an unfinished JSON edit survives; a model the dropdown does
  not list — one from List models, or a custom id — comes back as a custom entry with its
  client type. The API key is part of what is stored, so it stays in that browser profile
  until the field is cleared.

Alternatives weighed

• Raise instead of dropping the history. Honest, but it leaves the playground unusable
  after the first answer, and there is nothing the caller could do except clear the session
  before every message. The vendor treats a TTS call as one synthesis request, so the client
  sends one.
• Fix it in the playground only (call the stateless API there). That leaves
  streaming_response_stateful broken for TTS in both SDKs.
• Keep sending earlier user turns, minus the audio. Rejected by the API too: the 400 is
  about the turn count, not the modality.
• Rebuild the player on every chunk so the audio is playable while it streams. A data
  URL cannot be appended to, so each chunk would re-encode the whole clip (~190 rebuilds for
  a 7-second answer). Left the placeholder counting up instead.
• autoplay on the element instead of calling play(). The attribute fires again every
  time the card is re-parsed, which is the opposite of playing once.

Verification

• Two stateful turns with a system prompt and a thinking level set both succeed; transcribing
  the two clips back with gemini-3.7-flash returns each turn's own text, so the trimmed
  history does not leak into the audio.
• Both playgrounds driven in a real browser (Playwright): one player per answer, valid RIFF
  WAV, no console errors, and Stop mid-synthesis leaves one playable partial clip plus the
  Interrupted marker.
• Reload check in both playgrounds: the panel comes back field for field, including the
  combobox labels, an unlisted model restored as a custom entry with its client type, and no
  focus stolen into the custom id field on load.
• Autoplay check: the clip is at 1.4 s of its 2.24 s duration 1.5 s after the stream ends,
  with the token footer rendered beside it.
• A traced TTS call renders one <audio> element in the tracer instead of one per chunk.
• E2E: pytest -k tts (4 passed), pytest -k gemini-3.7-flash (11 passed), jest -t gemini-3.1-flash-tts-preview (14 passed), jest -t gemini-3.7-flash (20 passed).
• ruff check / ruff format --check, npm run lint, npm run build, and the offline
  suites in both languages are green.

Not covered

The multi-turn TTS path has no E2E assertion. Covering it means adding a second turn to the
shared test_tts_generation_single_speaker / its TS counterpart, which the dev skill asks
for explicit approval before touching.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UBwEdjegVGtfY2yJKwhZ77

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.