Noise and Voice Breaking in HeyGen LiveAvatar for Long Sentences

[bhavinmatariya]

Hello HeyGen team / Maintainers,
I am facing an issue while using the HeyGen LiveAvatar Web SDK.

Problem Description

• When the input sentence is short, the avatar speaks clearly without any issues.
• When the input sentence is long, the avatar output contains:
  • Noticeable noise
  • Voice breaking / distortion, as if the audio stream is getting corrupted or interrupted

⚠️ Important note:
The Text-to-Speech (TTS) output itself is clear and noise-free. The issue appears only after the audio is passed to the LiveAvatar SDK.

My Audio & Avatar Pipeline

User Text Input ("Hello")
↓
[TextInput Component]
↓
[useTextChat Hook]
↓

1. Session Initialization (if required)
   POST /api/custom-chat-initiate
   → AI Agent API
   → Returns session token (cookie)
   ↓
2. Send Message to AI Agent
   POST /api/custom-chat-message
   → WebSocket connection pool
   → AI Agent WebSocket (wsUrl)
   → Returns AI response text
   ↓
   (AI response text)
   ↓
3. Text-to-Speech Conversion
   POST /api/huggingface-text-to-speech
   → HuggingFace SBV2 API
   → Returns audio (base64)
   → FFmpeg converts audio to PCM (24kHz, mono, s16le)
   ↓
   (PCM audio base64)
   ↓
   LiveAvatarSession.repeatAudio()
   ↓
4. Avatar Speech
   → Audio split into chunks
   → Sent via WebSocket to HeyGen backend
   ↓
5. Avatar Video Stream
   → LiveKit Room (WebRTC)
   → Video track from HeyGen
   → Audio track from HeyGen
   → Lip-synced avatar video
   ↓
   Browser Video Element
   ↓
   User sees & hears avatar speaking

Observations

• ✅ TTS audio (before sending to LiveAvatar) is perfectly clear
• ❌ Noise and distortion occur only when the sentence is long
• ❓ Feels like:
  • Audio chunking issue
  • Buffer overflow / timing issue
  • Sample rate or frame alignment issue for long audio

Expected Behavior
The avatar should speak long sentences smoothly, just like short sentences, without introducing noise or voice distortion.

Additional Info

• I have attached a video recording showing:
• Noisy / broken avatar speech for long text

VIDEO URL :- https://drive.google.com/file/d/15Wo3kITyATEoOw2h3g_5mKnkzkzAvJM2/view?usp=drive_link

Any guidance or suggestions would be greatly appreciated.
Thanks in advance for your support.