Add realtime streaming to HTTP server

[carli2]

Add realtime streaming to HTTP server

Summary

Adds an optional realtime flag to the POST / synthesis endpoint in piper.http_server. When set, the server immediately sends a provisional WAV header (with an estimated data length), streams PCM frames as they are produced, and zero‑pads any remaining bytes to match the header length. Default behavior (buffered WAV response) is unchanged.

Motivation

• Lower end‑to‑end latency for clients that begin playback as soon as headers arrive.
• Preserve compatibility with simple WAV clients while enabling streaming.

Implementation

• New request field: realtime: true (boolean). No CLI changes; per‑request control.
• Provisional header: estimate total bytes using a simple chars/sec model, sentence count, configured --sentence-silence, a 0.5s margin, and a 1.5× safety factor.
• Stream frames as generated; insert inter‑sentence silence like the buffered path.
• Pad with zeros to reach the estimated data size. If actual audio exceeds the estimate (rare), log a warning and finish the stream.
• Files touched:
  • src/piper/http_server.py
  • docs/API_HTTP.md (document realtime field)

Usage

python -m piper.http_server -m de_DE-thorsten-medium --data-dir ~/projekte/hardlife
curl -X POST -H 'Content-Type: application/json' \
  -d '{"text":"Hallo Welt","realtime":true}' \
  -o out.wav localhost:5000

Testing

• Manual: verify immediate header arrival and valid WAV playback while data streams.
• Validate the final file opens in common players and that padding is silent tail data.
• Confirm default requests (no realtime) behave exactly as before.

Backwards Compatibility

• Opt‑in only; no behavior change for existing clients.

Follow‑ups (optional)

• Server flag for default realtime (e.g., --realtime-default).
• Configurable estimate constants (chars/sec, margin, safety factor).

Checklist

• Format/lint: script/format and script/lint
• Docs: docs/API_HTTP.md updated
• Local test requests (buffered and realtime)

[LinFei83]

Excellent improvement! Thank you for your contribution, real-time streaming output is very helpful for reducing latency.
I have a related idea: Could the text to be synthesized also support streaming input? This would allow for better adaptation to the streaming output of large language models (LLMs), achieving the lowest end-to-end latency.

[carli2]

Excellent improvement! Thank you for your contribution, real-time streaming output is very helpful for reducing latency. I have a related idea: Could the text to be synthesized also support streaming input? This would allow for better adaptation to the streaming output of large language models (LLMs), achieving the lowest end-to-end latency.

LLMs often stream JSON events per line. Would you suggest to support a similar JSONl input or would you stick to character based input?

[LinFei83]

Excellent improvement! Thank you for your contribution, real-time streaming output is very helpful for reducing latency. I have a related idea: Could the text to be synthesized also support streaming input? This would allow for better adaptation to the streaming output of large language models (LLMs), achieving the lowest end-to-end latency.

LLMs often stream JSON events per line. Would you suggest to support a similar JSONl input or would you stick to character based input?

Hi carli2,

Thanks for the quick reply and for being open to the idea!

That's a great question. I believe JSONL (JSON Lines) would be the more robust and flexible approach, for a few key reasons:

1. Alignment with LLMs: Many large language models stream their output as Server-Sent Events (SSEs), which often contain JSON objects. A JSONL input would align perfectly with this pattern, making integration very straightforward.
2. Extensibility: A structured format like JSON allows for easily adding metadata in the future. For example, we could include flags to signal sentence boundaries ("flush": true), request a speaker change, or even convey emotion, without breaking the protocol.
3. Clarity: It's more explicit than a raw character stream. The server wouldn't have to guess when a sentence or a synthesizable chunk of text is complete.

A potential stream could look something like this:

{"text": "Hello,"}
{"text": " this is"}
{"text": " a streaming test.", "flush": true}
{"text": " Here is another sentence."}

The server could buffer the text until it receives a line with "flush": true (or hits a natural sentence break), at which point it synthesizes the buffered audio.

A simple character-based stream is also viable, but I think the structure of JSONL offers significant long-term benefits.

Thanks again for your excellent work on this PR!