[FEAT] Voxtral Support

[jorge-menjivar]

• Adds support for mistralai/Voxtral-Mini-3B-2507.
• Should also support mistralai/Voxtral-Small-24B-2507, but this is not tested.

These models only have support for tekken tokenizer files right now, so I created and added an optional tekken tokenizer crate to the examples workspace. (tekken-rs). This gets enabled with the tekken feature.

fixes #3028

Run example with

cargo run --example voxtral --features tekken,symphonia,cuda,cudnn --release

[jorge-menjivar]

@maximizemaxwell Thank you for your earlier work! Here is the working code if you want to try it.

[maximizemaxwell]

@greenrazer Could you review this feature implementations?

[benedikt-schaber]

Hey, thank you for the implementation. I had a rough one myself but was missing the tekken part among some other flaws. I went through the code and I think I might have some pointers for improvement. I am new to audio transformers, however, so please have some leniency with any mistakes :)

For the example:

• unused code: the example has some unused code, it loads the mel filters when creating the model, but does not use them. Further, the reimplementation of pcm_to_filer in audo_processing.rs is unused
• audo_utils.rs only contains a resample function, pcm_decode is its own module, so I think this could be named more specifically
• seperating transcribe_audio_with_tokens from transcribe_audio_internal seems unecessary
• post_process_transcription seems very specific, especially with the word combinators, is this necessary / should this be generalized? I only did a small number of tests (around 100 audio files of 30secs each) without the post processing but I also did not encounter the issues there.

For the main code:

• voxtral_llama.rs might be mergable in llama.rs, it only adds the explicit head_dim option, voxtral defaults and "fixes" dtype consistency, single sequence support

[greenrazer]

Thank you for the contribution, great job!

Just fix the clippy errors and we should be good:

cargo clippy --workspace --tests --examples --fix -- -D warnings

[jorge-menjivar]

Thank you @greenrazer and @benedikt-schaber for the suggestions.

I have made the requested changes.

I have also tested whisper's and snac's examples to make sure they run correctly after moving the pcm_decode and resample functions to the shared module.

[greenrazer]

Looks good, Thanks!