Fine-tuning Turbo on new language (Norwegian): Posterior Collapse / Audio Reconstruction issue

[torsteinelv]

I am attempting to fine-tune Chatterbox Turbo (T3) on a new language. I have expanded the tokenizer and set up the training pipeline, but I am facing persistent issues where the model fails to align with the text.

Observed Behavior The model completely ignores the input text and behaves in two extreme ways depending on the sampling temperature:

Low Temperature: The model outputs silence (or immediately stops).

High Temperature: The model reproduces the reference audio prompt almost perfectly (acting like an autoencoder), but ignores the text content entirely. It does not speak the requested text.

Data & Experiments I have tested this with multiple dataset configurations, ranging from 10 hours up to 100 hours of high-quality speech data. The behavior remains consistent regardless of the dataset size.

Technical Setup

Model: t3_turbo_v1.safetensors

Modifications: I am removing the backbone WTE layer (del new_t3_model.tfmr.wte) before training, as per standard Turbo fine-tuning scripts.

Frozen Modules: Voice Encoder and S3Gen are frozen. Only T3 is training.

Hyperparameters: Batch size 6, Grad Accum 4.

Question Has anyone experienced this specific behavior where the model only learns to reconstruct the audio prompt? Are there specific hyperparameters or tricks required when fine-tuning Turbo on a completely new language to force it to attend to the text?

[Mareike-RTY]

I'm facing the same issue. Just finished finetuning Chatterbox-Turbo on a German dataset, and it either results in gibberish audio or - if I use a reference audio of a speaker unseen in training - it reconstructs the text spoken in the reference audio instead of the text I want it to speak.
I'm now conducting a second finetuning experiment, increasing the number of epochs from 150 to 500 as suggested in #6 (comment) (batch size 32, grad_accum 1, learning rate 1e-5). I hope that's going to solve this issue...

Edit: the inference sample created during training (InferenceCallback) also just recreates a text from the training data instead of the given inference_test_text, so I presume just increasing the number of epochs won't fix the reconstruction issue.

[Mareike-RTY]

@gokhaneraslan: First of all, thanks for your great work and effort. I enjoyed your detailed and well-documented finetuning guide. Do you have any suggestions regarding the problems described above? It's most appreciated.

[gokhaneraslan]

I suspect this is actually a deeper structural issue inherent to the original model architecture, specifically regarding how t3.py handles the WTE.
After analyzing the code and running several experiments, WTE Bypass, the T3 class in t3.py uses its own custom self.text_emb layer and explicitly bypasses the backbone transformer's internal WTE. When we extend the tokenizer for a new language (e.g., Arabic/German), these new token embeddings in self.text_emb are fresh. Even when we initialize them using the mean of existing tokens (to avoid instability), they provide a very "weak" or "blurry" signal compared to the pre-trained, powerful Audio Encoder.
And Collapse, Since the Audio signal is strong and the Text signal is weak/averaged, the model takes the path of least resistance: it ignores the text entirely and collapses into an autoencoder state (just reconstructing the reference audio to minimize loss).
I have tried different initialization strategies (including mean initialization), but the model still tends to ignore the text. It seems this requires a structural fix—likely implementing Audio Conditioning Dropout during training to force the model to attend to the text embeddings, rather than just relying on the reference audio.

[Malshan-svg]

Hi, I'm stuck at extending the tokenizer, please can somebody help me. should i manually update the tokenizer.json with the new language characters and change the new vocab size accordingly or is there a programmatic way to do that?

[gokhaneraslan]

Currently, you need to manually update the tokenizer.json file. There isn't a programmatic script in the repository yet that automatically scans your dataset and extends the vocabulary for completely new alphabets.

Here is the exact workflow you should follow to add a new language (if its characters aren't already in the default 23-language set):

Open pretrained_models/tokenizer.json in a text editor.

Add your new characters at the end of the JSON mapping. Make sure each new character gets a unique, sequential ID.

Count the exact total number of items in your updated JSON file.

Update NEW_VOCAB_SIZE: Put this new total number into both src/config.py AND inference.py.

Important Note for Turbo Mode:
If you are using Turbo Mode (is_turbo = True), you still need to manually add your characters to the base tokenizer.json first. After you do that, when you run python setup.py, the script will programmatically take your newly added characters and smartly merge them into the large GPT-2 BPE tokenizer. It will then print a message saying something like "Please update the 'new_vocab_size' in 'src/config.py' to: XXXXX". Just copy that number into your config, and you're good to go!

Writing a programmatic script to auto-generate the tokenizer.json from a dataset is actually a good feature idea, I might add a script for that in the future.