I build kannada language from scratch using kaldi with 60hr , how to finetune with 1hr?

[karthikeyapranav]

I build kannada model from scratch with kaldi all { lang.sh, n-gram for language_model.apra , lexicon with own g2p , mfcc , mono, tri 1,2,3 , ivector , chain nnet3 tdnn, dynamic graph with opengram } - it works good

but how i can fine tune with 1hr of data = i want to make model more powerfull to make branches and add misssing words which improve accuracy

I cant find perfect steps to follow? - where to find fine tune steps???

what i got is = { extract mfcc of 1hr , by using tri3 of 60hr align_fmllr_lats.sh send that and do ivector - } like its bit confusing onto their is no proper documentaction is their

please provide steps to finetune.

[nshmyrev]

Something like this script

#!/usr/bin/env bash

dir=exp/chain_finetune/cnn_tdnn_finetune

src_mdl=exp/chain/tdnn/final.mdl

src_tree_dir=exp/chain/tree

primary_lr_factor=0.25 # The learning-rate factor for transferred layers from source
                       # model. e.g. if 0, the paramters transferred from source model
                       # are fixed.
                       # The learning-rate factor for new added layers is 1.0.

phone_lm_scales="1,100" # comma-separated list of positive integer multiplicities
                        # to apply to the different source data directories (used
                        # to give the RM data a higher weight).

set -e -o pipefail
stage=0
nj=10
train_set=train
gmm=tri4
num_epochs=2

# The rest are configs specific to this script.  Most of the parameters
# are just hardcoded at this level, in the commands below.
train_stage=-10
tdnn_affix=
nnet3_affix=_finetune
common_egs_dir=
#dropout_schedule='0,0@0.20,0.5@0.50,0'
dropout_schedule='0,0@0.20,0.1@0.50,0'
remove_egs=true
xent_regularize=0.25
get_egs_stage=-10
# End configuration section.
echo "$0 $@"  # Print the command line for logging

. ./cmd.sh
. ./path.sh
. ./utils/parse_options.sh


train_data_dir=data/${train_set}
gmm_dir=exp/${gmm}
lat_dir=exp/chain_finetune/${gmm}_${train_set}_lats
lang_dir=data/lang_test
dir=exp/chain${nnet3_affix}/tdnn${tdnn_affix}
train_ivector_dir=exp/chain/ivectors_train


if [ $stage -le 4 ]; then

    steps/make_mfcc.sh --mfcc-config conf/mfcc.conf --nj 10 data/train exp/make_mfcc/train mfcc
    steps/compute_cmvn_stats.sh data/train exp/make_mfcc/train mfcc
    utils/fix_data_dir.sh data/train

    steps/online/nnet2/extract_ivectors_online.sh --nj 4 \
      data/train exp/chain/extractor \
      exp/chain/ivectors_train
fi

if [ $stage -le 5 ]; then
  steps/align_fmllr_lats.sh --nj $nj --cmd "$train_cmd" \
    --generate-ali-from-lats true \
  $train_data_dir $lang_dir $gmm_dir $lat_dir
  rm $lat_dir/fsts.*.gz
fi

if [ $stage -le 6 ]; then
  # Set the learning-rate-factor for all transferred layers but the last output
  # layer to primary_lr_factor.
  $train_cmd $dir/log/generate_input_mdl.log \
    nnet3-am-copy --raw=true --edits="set-learning-rate-factor name=* learning-rate-factor=$primary_lr_factor; set-learning-rate-factor name=output* learning-rate-factor=1.0" \
      $src_mdl $dir/input.raw || exit 1;
fi

if [ $stage -le 7 ]; then
  echo "$0: compute {den,normalization}.fst using weighted phone LM with icsi,ami and safet weight $phone_lm_scales."
  steps/nnet3/chain/make_weighted_den_fst.sh --cmd "$train_cmd" \
    --num-repeats $phone_lm_scales \
    --lm-opts '--num-extra-lm-states=200' \
    $src_tree_dir $lat_dir $dir || exit 1;
fi

if [ $stage -le 8 ]; then
  # exclude phone_LM and den.fst generation training stages
  if [ $train_stage -lt -4 ]; then train_stage=-4 ; fi
  steps/nnet3/chain/train.py --stage $train_stage \
    --cmd "$decode_cmd" \
    --trainer.input-model $dir/input.raw \
    --feat.online-ivector-dir "$train_ivector_dir" \
    --feat.cmvn-opts "--norm-means=false --norm-vars=false" \
    --chain.xent-regularize $xent_regularize \
    --chain.leaky-hmm-coefficient 0.1 \
    --chain.l2-regularize 0.0 \
    --chain.apply-deriv-weights false \
    --chain.lm-opts="--num-extra-lm-states=2000" \
    --egs.dir "$common_egs_dir" \
    --egs.opts "--frames-overlap-per-eg 0 --constrained false --num-utts-subset 50" \
    --egs.chunk-width 140,100,160 \
    --trainer.num-chunk-per-minibatch 64 \
    --trainer.frames-per-iter 3000000 \
    --trainer.num-epochs $num_epochs \
    --trainer.optimization.num-jobs-initial 1 \
    --trainer.optimization.num-jobs-final 1 \
    --trainer.optimization.initial-effective-lrate 0.00001 \
    --trainer.optimization.final-effective-lrate 0.00001 \
    --trainer.max-param-change 2.0 \
    --trainer.dropout-schedule $dropout_schedule \
    --trainer.add-option="--optimization.memory-compression-level=2" \
    --cleanup.remove-egs $remove_egs \
    --feat-dir $train_data_dir \
    --tree-dir $src_tree_dir \
    --lat-dir $lat_dir \
    --dir $dir  || exit 1;
fi

[karthikeyapranav]

This helps a lot , previously i thought we need to freeze old layera and only train last 2 , but the above code

Something like this script

#!/usr/bin/env bash

dir=exp/chain_finetune/cnn_tdnn_finetune

src_mdl=exp/chain/tdnn/final.mdl

src_tree_dir=exp/chain/tree

primary_lr_factor=0.25 # The learning-rate factor for transferred layers from source
                       # model. e.g. if 0, the paramters transferred from source model
                       # are fixed.
                       # The learning-rate factor for new added layers is 1.0.

phone_lm_scales="1,100" # comma-separated list of positive integer multiplicities
                        # to apply to the different source data directories (used
                        # to give the RM data a higher weight).

set -e -o pipefail
stage=0
nj=10
train_set=train
gmm=tri4
num_epochs=2

# The rest are configs specific to this script.  Most of the parameters
# are just hardcoded at this level, in the commands below.
train_stage=-10
tdnn_affix=
nnet3_affix=_finetune
common_egs_dir=
#dropout_schedule='0,0@0.20,0.5@0.50,0'
dropout_schedule='0,0@0.20,0.1@0.50,0'
remove_egs=true
xent_regularize=0.25
get_egs_stage=-10
# End configuration section.
echo "$0 $@"  # Print the command line for logging

. ./cmd.sh
. ./path.sh
. ./utils/parse_options.sh


train_data_dir=data/${train_set}
gmm_dir=exp/${gmm}
lat_dir=exp/chain_finetune/${gmm}_${train_set}_lats
lang_dir=data/lang_test
dir=exp/chain${nnet3_affix}/tdnn${tdnn_affix}
train_ivector_dir=exp/chain/ivectors_train


if [ $stage -le 4 ]; then

    steps/make_mfcc.sh --mfcc-config conf/mfcc.conf --nj 10 data/train exp/make_mfcc/train mfcc
    steps/compute_cmvn_stats.sh data/train exp/make_mfcc/train mfcc
    utils/fix_data_dir.sh data/train

    steps/online/nnet2/extract_ivectors_online.sh --nj 4 \
      data/train exp/chain/extractor \
      exp/chain/ivectors_train
fi

if [ $stage -le 5 ]; then
  steps/align_fmllr_lats.sh --nj $nj --cmd "$train_cmd" \
    --generate-ali-from-lats true \
  $train_data_dir $lang_dir $gmm_dir $lat_dir
  rm $lat_dir/fsts.*.gz
fi

if [ $stage -le 6 ]; then
  # Set the learning-rate-factor for all transferred layers but the last output
  # layer to primary_lr_factor.
  $train_cmd $dir/log/generate_input_mdl.log \
    nnet3-am-copy --raw=true --edits="set-learning-rate-factor name=* learning-rate-factor=$primary_lr_factor; set-learning-rate-factor name=output* learning-rate-factor=1.0" \
      $src_mdl $dir/input.raw || exit 1;
fi

if [ $stage -le 7 ]; then
  echo "$0: compute {den,normalization}.fst using weighted phone LM with icsi,ami and safet weight $phone_lm_scales."
  steps/nnet3/chain/make_weighted_den_fst.sh --cmd "$train_cmd" \
    --num-repeats $phone_lm_scales \
    --lm-opts '--num-extra-lm-states=200' \
    $src_tree_dir $lat_dir $dir || exit 1;
fi

if [ $stage -le 8 ]; then
  # exclude phone_LM and den.fst generation training stages
  if [ $train_stage -lt -4 ]; then train_stage=-4 ; fi
  steps/nnet3/chain/train.py --stage $train_stage \
    --cmd "$decode_cmd" \
    --trainer.input-model $dir/input.raw \
    --feat.online-ivector-dir "$train_ivector_dir" \
    --feat.cmvn-opts "--norm-means=false --norm-vars=false" \
    --chain.xent-regularize $xent_regularize \
    --chain.leaky-hmm-coefficient 0.1 \
    --chain.l2-regularize 0.0 \
    --chain.apply-deriv-weights false \
    --chain.lm-opts="--num-extra-lm-states=2000" \
    --egs.dir "$common_egs_dir" \
    --egs.opts "--frames-overlap-per-eg 0 --constrained false --num-utts-subset 50" \
    --egs.chunk-width 140,100,160 \
    --trainer.num-chunk-per-minibatch 64 \
    --trainer.frames-per-iter 3000000 \
    --trainer.num-epochs $num_epochs \
    --trainer.optimization.num-jobs-initial 1 \
    --trainer.optimization.num-jobs-final 1 \
    --trainer.optimization.initial-effective-lrate 0.00001 \
    --trainer.optimization.final-effective-lrate 0.00001 \
    --trainer.max-param-change 2.0 \
    --trainer.dropout-schedule $dropout_schedule \
    --trainer.add-option="--optimization.memory-compression-level=2" \
    --cleanup.remove-egs $remove_egs \
    --feat-dir $train_data_dir \
    --tree-dir $src_tree_dir \
    --lat-dir $lat_dir \
    --dir $dir  || exit 1;
fi

This helps a lot , previously i thought we need to freeze old layera and only train last 2 , but the above code is doing soft freeze the input layesa will get 0.25 learning rate and last layesa will get 1.0 so no old knowladge will be vomited that much and still carry those weights and biases.

[nshmyrev]

Ok, let us know how it goes

[karthikeyapranav]

Hi again thanks for the script and a quick follow-up on the fine-tuning approach. The 60hr to 61hr (60+1hr) round worked great, 20% WER improvement, using your finetune.sh script as the base.

For the next round, I tried fine-tuning that already-fine-tuned 61hr model with another 1hr batch reusing the same setup (same tree, same script structure, source model = the previous round's final.mdl [61hr] instead of a fresh 60hr model):

cp -r $BUILD61/data/lang/* $NEW/data/lang/
cp -r $BUILD61/data/local $NEW/data/local_60hr
cp -r $BUILD61/exp/tri3b/* $NEW/exp/tri3b/
cp -r $BUILD61/exp/chain/tree $NEW/exp/chain/tree
cp -r $BUILD61/exp/chain_finetune/final_model/* $NEW/exp/chain/tdnn/
cp -r $BUILD61/exp/chain/extractor $NEW/exp/chain/extractor
cp -r $BUILD61/exp/chain/diag_ubm $NEW/exp/chain/diag_ubm
cp $BUILD61/conf/mfcc.conf $NEW/conf/mfcc.conf

This time accuracy didn't improve (roughly flat/slightly worse) and we also started seeing RAM spikes/decoder slowdowns on some audio

So My Question =
is repeatedly fine-tuning a model that's already been fine-tuned (61hr fine-tuned → fine-tune again with +1hr +1hr → ...) expected to work ?with the same tree/GMM/extractor held fixed, the way it does in a single 60hr→61hr round? Or is there a point where the frozen tree/alignments (built from the original 60hr GMM) stop being a good match as the model moves further from that original base, and a fresh full retrain (GMM + tree) is needed before continuing

Basically is chained fine-tuning (60→61→62→63...) safe to do with primary_lr_factor=0.25, or does it need periodic "reset" retrains as the gap from the original tree grows

[nshmyrev]

Its a waste of time to collect data hour by hour. You need 1000 hours at least, there are datasets on AI4Bharat, you can use them

https://huggingface.co/datasets/ai4bharat/IndicVoices/tree/main/kannada

[karthikeyapranav]

Thanks for the pointer we'll move to using AI4Bharat's larger sets (Kathbath, IndicVoices, Shrutilipi) instead of building up hour-by-hour

Request: For the Telugu Vosk model on the models page, we're planning to fine-tune it with more data. To do that we'd need:

*data/lang/
*data/local
*exp/tri3b
*exp/chain/tree
*exp/chain/tdnn
*exp/chain/extractor
*exp/chain/diag_ubm

Could you share these training artifacts, or alternatively the dataset(s) used to build the Telugu model, so we can reproduce the same tree/lexicon/extractor setup for fine-tuning?

Small doubt: After our 60hr→61hr fine-tune (worked well, +20% WER), we tried fine-tuning that same 61hr model again with another 1hr — same frozen tree/extractor/GMM, same primary_lr_factor=0.25 as round 1.

This time accuracy stayed flat/regressed and RAM usage went up during decoding on some files. I already tried adjusting the epoch count and initial/final learning rate in the training script, but it still didn't fix it — wondering if the 0.25 soft-freeze itself needs to change for a second round,

or if there's a known issue with fine-tuning an already-fine-tuned model repeatedly. Is there a rule of thumb for how much new data you need relative to current model size before chaining another round, or do we need a periodic full retrain (fresh GMM+tree) at some point regardless?