report.tex

\documentclass[11pt]{article}

% Use "final" for final version, "review" for review version, or "preprint" for non-anonymous version with page numbers.
\usepackage[final]{acl}

% Standard package includes
\usepackage{times}
\usepackage{latexsym}

% For proper rendering and hyphenation of words containing Latin characters (including in bib files)
\usepackage[T1]{fontenc}
% For Vietnamese characters
% \usepackage[T5]{fontenc}
% See https://www.latex-project.org/help/documentation/encguide.pdf for other character sets

% This assumes your files are encoded as UTF8
\usepackage[utf8]{inputenc}

% This is not strictly necessary, and may be commented out,
% but it will improve the layout of the manuscript,
% and will typically save some space.
% \usepackage{microtype}  % Commented out - requires additional fonts

% This is also not strictly necessary, and may be commented out.
% However, it will improve the aesthetics of text in
% the typewriter font.
% \usepackage{inconsolata}  % Commented out - install if needed

%Including images in your LaTeX document requires adding
%additional package(s)
\usepackage{graphicx}

% If the title and author information does not fit in the area allocated, uncomment the following
%
%\setlength\titlebox{<dim>}
%
% and set <dim> to something 5cm or larger.

\title{Project Handover Report: [Training Small Audio Embeddings]}

\begin{document}
\maketitle
\begin{abstract}
This document serves as a comprehensive handover report for training audio embedding models project. 
It provides an overview of the project's objectives, current status and possible directions to move forward.
This report is intended to facilitate a smooth transition to the next person responsible for this project.
\end{abstract}

\section{Project Goal}
The goal of this project is to train small audio embedding models that can be used for various audio retrieval tasks, 
either for audio-to-text retrieval or text-to-audio retrieval.
There are already some popular small audio embedding models available since 2022 based on the idea of contrastive learning \cite{CLAP2022,laionclap2023}.
This project aims to adopt existing multi-modal large language models (MLLMs) that supports audio modality as backbone 
and train a small audio embedding model that can be used for downstream retrieval tasks.

The core question is that we only have limited choices of mllm backbone (e.g. 3B, 7B), while we still want much smaller audio embedding models (e.g. 1.2B).
\textbf{Hence, the main goal is to train an audio embedding model smaller than 1.2B parameters, and still achieve comparable or 
better performance than the existing small audio embedding models.}

\section{Related Work}

\textbf{Contrastive Language-Audio Pretraining (CLAP)}
\citet{CLAP2022} extends the CLIP paradigm to the audio domain by learning audio concepts from natural language supervision.
The original CLAP model by \citet{CLAP2022} was trained with 128k audio-text pairs and evaluated on 16 downstream tasks across 8 domains, including sound event classification, music tasks, and speech-related tasks.
The model consists of an audio encoder and a text encoder (typically BERT-based) that project their outputs into a joint multimodal space using contrastive learning.

\citet{laionclap2023} scales up the approach by introducing LAION-Audio-630K, a large collection of 633,526 audio-text pairs from diverse sources.
The model incorporates two key innovations: (1) a feature fusion mechanism that enables processing audio inputs of variable lengths, and (2) keyword-to-caption augmentation to enhance training.
LAION-CLAP uses HTS-AT \cite{htsatke2022} (Hierarchical Token-Semantic Audio Transformer) as the audio encoder, which provides hierarchical audio representations.

\citet{CLAP2023} further advanced the CLAP approach by pretraining with a significantly larger and more diverse collection of 4.6M audio-text pairs.
This work introduces two key architectural innovations: (1) an audio encoder trained on 22 diverse audio tasks rather than just sound event classification, enabling richer audio representations, and (2) an autoregressive decoder-only language model instead of the standard encoder-only models (e.g., BERT), which better captures language semantics.
The model builds up sota on 26 downstream audio tasks, and is also the clap model that we compare against in our project.

\textbf{Recent advances in audio embedding models}
\citet{ma2025tevatron20unifieddocument} introduces Tevatron 2.0 where they finetune the Qwen2.5-omni model on the AudioCaps dataset.
 The model achieves competitive performance on AudioCaps evaluation set, but does not generalize well to other datasets that are not trained on.
 \citet{colqwenomniv01} introduces ColQwen-Omni-v0.1 based on Qwen2.5-Omni-3B. Noticeably, they do not exploit any audio-text pairs for training.
 Instead, they finetune the model purely on visual-document retrieval tasks, and rely on the prealigned LLMs to automatically generalize the embeddings to other modalities.
 Similar methods are also adopted in \citet{xu2025omniembednemotronunifiedmultimodalretrieval}. 
 Note that the work mentioned above uses different training data and evaluation metrics, so it is hard to directly compare the performance.

\section{Datasets}
\label{sec:datasets}
\subsection{Dataset used in this project}
Dataset is one of the most important factors that affects the model performance. 
In this project, we mainly adopted the following datasets:

\begin{itemize}
  \item \textbf{Clotho} (19,195 samples): An audio captioning dataset containing audio clips and their corresponding captions. 
  Available at \href{https://zenodo.org/record/3490684}{Zenodo}.
    
  \item \textbf{AudioSetStrong} (108,311 samples): Temporally-strong labeled events within 10s audio clips from an unofficial WavCaps version with GPT-generated sentences instead of simple text phrases. The audio data is the same as the original AudioSet Strong dataset.
  Available at \href{https://huggingface.co/datasets/cvssp/WavCaps}{HuggingFace}.
  
  \item \textbf{FSD50K} (40,966 samples): An open dataset of human-labeled sound events containing 51,197 Freesound clips unequally distributed across 200 classes drawn from the AudioSet Ontology.
  Available at \href{https://zenodo.org/record/4060432}{Zenodo}.
  
  \item \textbf{MACS} (3,930 samples): Audio captions and corresponding audio tags for the TAU Urban Acoustic Scenes 2019 development dataset (airport, public square, and park).
  Available at \href{https://huggingface.co/datasets/milesway6/MACS}{HuggingFace}.
  \textit{Note: This dataset includes the MACS test set in training data, making evaluation on MACS meaningless for our experiments.}
  
  \item \textbf{UrbanSound8K} (8,732 samples): Urban sound classification dataset containing labeled sound excerpts from urban environments.
  Available at \href{https://github.com/LAION-AI/audio-dataset}{GitHub}.
\end{itemize}

\subsection{Datasets to collect}
Besides the datasets above, some previous work also includes other datasets for training. 
Future work can also consider collecting them for larger scale training:

\begin{itemize}
  \item \textbf{CMU Arctic}: Speech synthesis database with single-speaker recordings. \href{http://festvox.org/cmu_arctic/}{[Link]}
  
  \item \textbf{FLEURS}: Multilingual speech dataset covering 102 languages. \href{https://huggingface.co/datasets/google/fleurs}{[Link]}
  
  \item \textbf{MusicNet}: Classical music dataset with note-level annotations. \href{https://github.com/LAION-AI/audio-dataset/blob/main/download_script/download_musicnet.sh}{[Link]}
  
  \item \textbf{WavCaps}: Large-scale audio captioning dataset with diverse sound sources. \href{https://huggingface.co/datasets/cvssp/WavCaps}{[Link]}
  
  \item \textbf{MusicCaps}: Music audio captioning dataset from Google. \href{https://huggingface.co/datasets/google/MusicCaps}{[Link]}
  
  \item \textbf{CLAP Freesound}: Curated audio dataset from Freesound for CLAP training. \href{https://huggingface.co/datasets/Meranti/CLAP_freesound}{[Link]}
  
  \item \textbf{NSynth}: Large-scale audio dataset of musical notes from Google Magenta. \href{https://magenta.withgoogle.com/datasets/nsynth}{[Link]}
  
  \item \textbf{Audio Retrieval Benchmark}: Comprehensive benchmark for audio retrieval tasks. \href{https://github.com/akoepke/audio-retrieval-benchmark}{[Link]}
  
  \item \textbf{SoundDescs}: Audio captioning dataset. \href{https://zenodo.org/records/7080122}{[Link]}
  
  \item \textbf{EPIC-KITCHENS}: Egocentric video dataset with audio from kitchen activities. \href{https://github.com/epic-kitchens/epic-kitchens-download-scripts}{[Link]}
  
  \item \textbf{LJ Speech}: Public domain speech dataset for text-to-speech research. \href{https://keithito.com/LJ-Speech-Dataset/}{[Link]}
  
  \item \textbf{AudioCaps (Tevatron)}: AudioCaps variants prepared for retrieval tasks. \href{https://huggingface.co/datasets/Tevatron/audiocaps}{[Link]}
  
  \item \textbf{MELD}: Multimodal emotion recognition dataset with audio and video. \href{https://affective-meld.github.io/}{[Link]}
\end{itemize}


\begin{table*}[t]
  \centering
  \scriptsize
  \begin{tabular}{llcccccccccccc}
    \hline
    \textbf{Model} & \textbf{Config} & \multicolumn{2}{c}{\textbf{AudioCaps A2T}} & \multicolumn{2}{c}{\textbf{AudioCaps T2A}} & \multicolumn{2}{c}{\textbf{AudioSetStrong A2T}} & \multicolumn{2}{c}{\textbf{AudioSetStrong T2A}} & \multicolumn{2}{c}{\textbf{Clotho A2T}} & \multicolumn{2}{c}{\textbf{Clotho T2A}} \\
    & & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 \\
    \hline
    \multicolumn{14}{l}{\textit{Baseline Models}} \\
    \tiny \citet{CLAP2023} & --- & 4.0 & 46.4 & 17.2 & 42.0 & \textbf{25.4} & \textbf{58.6} & \textbf{16.0} & \textbf{44.3} & \textbf{5.3} & \textbf{55.3} & \textbf{16.0} & \textbf{39.0} \\
    \tiny \citet{ma2025tevatron20unifieddocument} & --- & \textbf{5.6} & \textbf{55.8} & \textbf{30.6} & \textbf{61.2} & 2.2 & 10.6 & 6.6 & 18.8 & 1.2 & 16.9 & 4.2 & 11.9 \\
    \tiny \citet{colqwenomniv01} & --- & 2.0 & 25.0 & 14.5 & 37.4 & 2.3 & 10.0 & 5.9 & 14.5 & 0.7 & 8.4 & --- & --- \\
    \hline
    \multicolumn{14}{l}{\textit{Our Models}} \\
    Qwen-3B & No LoRA, BS=32 & --- & --- & --- & 42.2 & --- & --- & --- & --- & --- & --- & --- & 35.8 \\
    Qwen-3B & LoRA, BS=32 & --- & --- & --- & 43.3 & --- & --- & --- & --- & --- & --- & --- & 33.5 \\
    Qwen-7B & No LoRA, BS=16 & --- & --- & --- & {47.7} & --- & --- & --- & --- & --- & --- & --- & {37.6} \\
    \hline
  \end{tabular}
  \caption{Performance comparison of baseline models and our finetuned Qwen2.5-Omni models on audio retrieval tasks. Baseline models are evaluated using our pipeline. Our models were only trained on Clotho. BS = batch size.}
  \label{tab:exp1-results}
\end{table*}

\section{Experiments We Have Done}
We have done the following experiments. Note that they might not be comparaible across different experiments due to different training data.
We didn't manage to perform exhaustive experiments because we wanted to quickly iterate on experiments and make decisions based on the results.

\begin{itemize}
  \item Experiment 1: Finetune a Qwen2.5-Omni-3B and 7B model on Clotho training set as baseline models.
  \item Experiment 2: Finetune the Qwen2.5-Omni-7B model with layer pruning to reduce the model size.
  \item Experiment 3: Finetune the Qwen2.5-Omni-7B model with modality transfer to see if the model trained on text pairs can generalize to other modalities.
  \item Experiment 4: Finetune the Qwen2.5-Omni-7B model with module combination to see if combining smaller modules (e.g.\ audio encoder, LLM) from 
  different models can achieve good performance.
\end{itemize}

\subsection{Evaluation setup}
We use consistent evaluation setup for all experiments.

\paragraph{Evaluation metrics.}
We use CV-Recall@5 (cvR@5) as the evaluation metric for text-to-audio retrieval tasks.
This is a binary version of recall that counts 1 if any correct item appears in the top-5 results, and 0 otherwise, averaged over all queries.

\paragraph{Evaluation datasets.}
The evaluation is performed on AudioCaps, AudioSetStrong, and Clotho from the audio branch in the MTEB codebase.
Note that MTEB codebase also support evaluation on UrbanSound8K and MACS, but we did not include them in the results.
For UrbanSound8K, it seems that at the time of writing, the text captions are not avaialble in the dataset, and so we can only get performance close to random.
For MACS, we did not include it in the results because in lots of our experiments we included the whole dataset (including test set) in the training data.
This makes evaluation on MACS meaningless for our experiments.

Also note that existing models are trained on different training sets, making direct comparison challenging.
For example, \citet{CLAP2023} is trained on both Clotho and AudioCaps training sets, whereas our models are only trained on Clotho.

\subsection{Experiment 1: Baseline Models with Qwen2.5-Omni}

\paragraph{Setup.}
We finetune Qwen2.5-Omni-3B and 7B models as baseline models.
We focus on the 7B variant for future experiments considering the license issue with the 3B model.
For training data, we use only the Clotho training set.

\paragraph{Results.}
Table~\ref{tab:exp1-results} shows the comprehensive performance comparison of baseline models and our finetuned models on audio retrieval tasks.
\citet{CLAP2023} achieves much higher performance on Clotho (39.0 cvR@5) compared to other models, which is not surprising since they are trained on 4.6M audio-text pairs including both Clotho and AudioCaps training sets.
\citet{ma2025tevatron20unifieddocument} performs best on AudioCaps T2A (61.2 cvR@5), which is the only dataset they are trained on.
\citet{colqwenomniv01} shows the lowest performance on AudioCaps (37.4 cvR@5) due to zero usage of audio data during training, but surprisingly the performance is not bad. 
This might suggest that as long as the MLLM aligns well with another modality (e.g., audio), it is able to generalize to it despite not being trained on data from that modality.

Our results on the Qwen models show that even with little training data (i.e., Clotho), the models can achieve competitive performance on AudioCaps and Clotho. 
Note that we did not utilize large-scale training data or perform full or LoRA finetuning here, since we expected to work with small models anyway.
The results for these large models are provided for reference only.

\subsection{Experiment 2: Layer Pruning for Model Size Reduction}

\begin{table*}[t]
  \centering
  \scriptsize
  \begin{tabular}{p{1.8cm}lcccccccccccc}
    \hline
    \textbf{Model} & \textbf{\# P} & \multicolumn{2}{c}{\textbf{AudioCaps A2T}} & \multicolumn{2}{c}{\textbf{AudioCaps T2A}} & \multicolumn{2}{c}{\textbf{AudioSetStrong A2T}} & \multicolumn{2}{c}{\textbf{AudioSetStrong T2A}} & \multicolumn{2}{c}{\textbf{Clotho A2T}} & \multicolumn{2}{c}{\textbf{Clotho T2A}} \\
    & & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 \\
    \hline
    \multicolumn{14}{l}{\textit{Baseline Models}} \\
    \tiny \citet{CLAP2023} & 250M & 4.0 & 46.4 & 17.2 & 42.0 & \textbf{25.4} & \textbf{58.6} & {16.0} & \textbf{44.3} & \textbf{5.3} & \textbf{55.3} & {16.0} & {39.0} \\
    \tiny \citet{ma2025tevatron20unifieddocument} & 7B & 5.6 & 55.8 & {30.6} & {61.2} & 2.2 & 10.6 & 6.6 & 18.8 & 1.2 & 16.9 & 4.2 & 11.9 \\
    \tiny \citet{colqwenomniv01} & 3B & 2.0 & 25.0 & 14.5 & 37.4 & 2.3 & 10.0 & 5.9 & 14.5 & 0.7 & 8.4 & --- & --- \\
    \hline
    \multicolumn{14}{l}{\textit{Our Models (Layer Pruning)}} \\
    5 layers & 2.3B & --- & --- & 25.4 & 56.0 & --- & --- & 19.3 & 47.9 & --- & --- & 14.5 & 36.0 \\
    10 layers & 3.5B & 7.9 & {66.5} & 28.3 & 58.2 & 21.1 & 52.0 & \textbf{20.9} & \textbf{50.6} & 4.2 & 48.5 & 14.2 & 36.5 \\
    20 layers & 5.8B & \textbf{8.2} & \textbf{70.1} & \textbf{32.2} & \textbf{63.2} & 20.9 & 48.1 & 20.7 & 49.0 & {4.8} & 49.5 & \textbf{16.4} & \textbf{39.2} \\
    \hline
  \end{tabular}
  \caption{Performance comparison on audio retrieval tasks with different layer configurations. Baseline models are evaluated using our pipeline. \# P = number of parameters.}
  \label{tab:layer-pruning}
\end{table*}


\paragraph{Setup.}
We investigate layer pruning as a method to reduce the model size while maintaining competitive performance.
Starting from Qwen2.5-Omni-7B, we experiment with using different numbers of transformer layers (5, 10, and 20 layers) to create models ranging from 2.3B to 5.8B parameters.
Training is performed for a maximum of 20,000 batches.

\paragraph{Training Data.}
We significantly expand the training data compared to Experiment 1, using a total of 181k audio-text pairs from five datasets (Clotho, AudioSetStrong, FSD50K, MACS, and UrbanSound8K as detailed in Section~\ref{sec:datasets}).

\paragraph{Results: Layer Pruning Effect.}
Table~\ref{tab:layer-pruning} compares the performance of our layer-pruned models against existing baselines across all evaluation datasets.
First, pruning layers does not significantly hurt performance on most tasks. Comparing the 5-layer model with the 20-layer model, 
the performance drops by about 7\% on AudioCaps and 2-3\% on other tasks.
Second, compared to the baseline models, our models are able match or exceed their performance on most tasks like AudioCaps and AudioSetStrong T2A.
However, even the 20-layer model can only match the performance of \citet{CLAP2023} on Clotho, which is likely due to the lack of enough training data.

Overall, it seems promising to use layer pruning to reduce the model size while maintaining competitive performance. 
However, it would be hard to further reduce the model size to 1B parameters by simply pruning depth while maintaining the width of the model.
We did not perform further experiments on layer pruning, but it would be interesting to see if further pruning width can still maintain competitive performance.

\paragraph{Results: Batch Size Effect.}
We also performed experiments to see the effect of batch size with the 10-layer model.
Table~\ref{tab:batch-size} shows the results.
Overall, test performance does not vary significantly with different batch sizes (32, 64, 128) on AudioCaps and Clotho.
Analysis of validation performance during training reveals that larger batch sizes help initially, but models with larger batches may degrade as training proceeds.
For example, on ClothoT2A, batch size 128 achieves 31.3 NDCG at 2,000 steps but drops to 29.3 at 10,000 steps.

\begin{table}[h]
  \centering
  \scriptsize
  \begin{tabular}{ccccccc}
    \hline
    \textbf{BS} & \multicolumn{2}{c}{\textbf{AudioCaps A2T}} & \multicolumn{2}{c}{\textbf{AudioCaps T2A}} & \multicolumn{2}{c}{\textbf{Clotho T2A}} \\
    & R@1 & R@5 & R@1 & R@5 & R@1 & R@5 \\
    \hline
    32 & \textbf{7.9} & 66.5 & 28.3 & 58.2 & 14.2 & \textbf{36.5} \\
    64 & 7.3 & 66.0 & 27.3 & 57.7 & 14.1 & 35.2 \\
    128 & 7.8 & \textbf{68.2} & \textbf{29.0} & \textbf{59.0} & \textbf{15.2} & 35.8 \\
    \hline
  \end{tabular}
  \caption{Effect of batch size on the 10-layer model (3.5B parameters). BS = batch size. Batch size 128 results from checkpoint at 12k steps.}
  \label{tab:batch-size}
\end{table}

\subsection{Experiment 3: Modality Transfer}
\paragraph{Motivation.}
\citet{xiao2025scalinglanguagecentricomnimodalrepresentation} proposes that MLLM latently learns alignment across modalities, and the models rely on information from other
modalities to generate text-only outputs.
Hence, simply finetuning the MLLM on the text-only data should be able to obtain good embeddings for other modalities. 
In their setup, they adopted NLI datasets for the text-only finetuning, and found that the models can achieve very good performance on audio modality.

\paragraph{Setup.}
Here we also want to study if the model can achieve good performance on other modalities by simply finetuning Qwen2.5-Omni-7B on the text-only data.
We use the same model as the layer-pruned 10-layer model (3.5B parameters), as well as the full Qwen2.5-Omni-7B model, and finetune the models on text-only datasets.
For the training data, we use five Natural Language Inference (NLI) and fact verification datasets: MultiNLI, SNLI, FEVER-fixed, FEVER argument pairs, and SciFact.
All datasets are equally weighted with no hard negatives used.
We finetune the model for 10,000 steps.

\begin{table*}[t]
  \centering
  \scriptsize
  \begin{tabular}{lcccccccccccc}
    \hline
    \textbf{Model} & \multicolumn{2}{c}{\textbf{Clotho A2T}} & \multicolumn{2}{c}{\textbf{Clotho T2A}} & \multicolumn{2}{c}{\textbf{AudioSetStrong A2T}} & \multicolumn{2}{c}{\textbf{AudioSetStrong T2A}} & \multicolumn{2}{c}{\textbf{AudioCaps A2T}} & \multicolumn{2}{c}{\textbf{AudioCaps T2A}} \\
    & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 \\
    \hline
    text\_pretrain\_nli & 1.0 & 18.1 & 1.2 & 4.4 & 10.9 & 30.9 & 2.3 & 5.1 & 1.5 & 26.8 & 2.1 & 5.9 \\
    text\_pretrain\_nli\_full & \textbf{2.6} & \textbf{33.0} & \textbf{11.3} & \textbf{28.5} & \textbf{13.3} & \textbf{35.7} & \textbf{16.8} & \textbf{40.8} & \textbf{4.1} & \textbf{47.5} & \textbf{20.2} & \textbf{46.1} \\
    \hline
  \end{tabular}
  \caption{Performance of models finetuned on text-only NLI datasets. Full finetuning significantly outperforms layer-pruned training and demonstrates cross-modal transfer to audio tasks.}
  \label{tab:text-only-training}
\end{table*}

\paragraph{Results.}
Table~\ref{tab:text-only-training} shows the performance of models finetuned on text-only NLI datasets.
The full finetuning approach significantly improves performance across all datasets, indicating that text-only finetuning can be a very promising method for audio embedding training.
This leads to two future directions: (1) Further experiments should explore continual training from the text-only finetuned checkpoint on audio-text data to determine whether performance can be further improved.
(2) We should investigate whether text-only finetuning can be applied to other text-only tasks beyond NLI. If successful, this would enable the use of much larger-scale datasets for training audio embedding models.
Notably, text-only finetuning does not work for layer-pruned models. This may be because layer-pruned models have not learned the audio-text alignment as well as the full model.

\subsection{Experiment 4: Module Combination for Model Size Reduction}

\paragraph{Motivation.}
The Qwen2.5-Omni series only provides limited choices for small models (the smallest being 3B).
We investigate whether combining independent modules (audio encoder and LLM) from different training stages can produce effective models smaller than 3B.

According to the Qwen2.5-Omni technical report, the model undergoes three training stages: 
\begin{itemize}
    \item \textbf{Stage 1:} Training vision and audio encoders with a frozen LLM using audio-text and image-text pairs.
    \item \textbf{Stage 2:} Unfreezing all parameters for a wider range of multimodal learning.
    \item \textbf{Stage 3:} Training with 32k sequence length for long-sequence understanding.
\end{itemize}

\paragraph{Setup.}
We design four model configurations (Table~\ref{tab:module-combination-configs}) that combine modules from different training stages to measure the transfer ability when audio encoders and LLMs are not trained jointly.
Each configuration uses modules from particular pretraining stages:
\begin{itemize}
    \item \textbf{M1}: uses an unadapted audio encoder (approximating pre-stage-1) with Qwen2.5-0.5B LLM (after stage 1). 
    \item \textbf{M2}: uses the same audio encoder as M1 with Qwen2.5-3B LLM.
    \item \textbf{M3}: uses the adapted Qwen2.5-Omni-3B audio encoder (after final stage) with Qwen2.5-3B LLM (after stage 1).
    \item \textbf{M4}: is the full Qwen2.5-Omni-3B model (all stages completed).
\end{itemize}
We use the same training data as Experiment 2 (181k samples total, detailed in Section~\ref{sec:datasets}).

\begin{table*}[h]
  \centering
  \scriptsize
  \begin{tabular}{lllllc}
    \hline
    \textbf{Model} & \textbf{Audio Encoder} & \textbf{Audio Stage} & \textbf{LLM} & \textbf{LLM Stage} & \textbf{\# P} \\
    \hline
    M1 & Qwen3-Omni-30B (0.6B) & Pre-stage-1 & Qwen2.5-0.5B & After stage 1 & 1.1B \\
    M2 & Qwen3-Omni-30B (0.6B) & Pre-stage-1 & Qwen2.5-3B & After stage 1 & 3.6B \\
    M3 & Qwen2.5-Omni-3B (0.8B) & After stage 3 & Qwen2.5-3B & After stage 1 & 3.8B \\
    M4 & Qwen2.5-Omni-3B (0.8B) & After stage 3 & Qwen2.5-Omni-3B & After stage 3 & 3.8B \\
    \hline
  \end{tabular}
  \caption{Model configurations for module combination showing pretraining stages. M1/M2 use unadapted audio encoder from Qwen3. M3/M4 use adapted audio encoder from Qwen2.5-Omni-3B. \# P = total parameters.}
  \label{tab:module-combination-configs}
\end{table*}

\paragraph{Results.}
Table~\ref{tab:module-combination} shows the performance of all four module combination configurations.
According to the table, the performance generally improves in the order M1 $<$ M2 $<$ M3 $<$ M4, demonstrating that using better-aligned and larger modules improves embedding performance.
Specifically, using LLM after stage 1 only (M3 vs. M4) does not significantly hurt performance on most datasets, suggesting that stages 2-3 of LLM pretraining may be less critical for embedding tasks.
Moreover, using an unaligned audio encoder (M2 vs. M3) degrades performance but does not completely corrupt the model, indicating some transferability across model versions.
Finally, LLM size has a substantial effect: switching from 3B to 0.5B LLM (M1 vs. M2) reduces performance across all datasets.

\begin{table*}[t]
  \centering
  \scriptsize
  \begin{tabular}{lcccccccccccc}
    \hline
    \textbf{Model} & \multicolumn{2}{c}{\textbf{Clotho A2T}} & \multicolumn{2}{c}{\textbf{Clotho T2A}} & \multicolumn{2}{c}{\textbf{AudioSetStrong A2T}} & \multicolumn{2}{c}{\textbf{AudioSetStrong T2A}} & \multicolumn{2}{c}{\textbf{AudioCaps A2T}} & \multicolumn{2}{c}{\textbf{AudioCaps T2A}} \\
    & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 \\
    \hline
    M1 & 4.0 & 42.7 & 10.6 & 28.8 & 18.8 & 47.1 & 15.6 & 40.2 & 7.2 & 61.8 & 22.2 & 49.7 \\
    M2 & 4.1 & 46.3 & 13.7 & 35.1 & 21.5 & 50.2 & 19.5 & 50.8 & 7.5 & 66.8 & 29.0 & 59.3 \\
    M3 & \textbf{4.8} & 49.0 & 15.9 & 38.5 & 21.7 & \textbf{55.7} & 21.7 & 51.2 & 8.1 & 68.5 & 31.4 & 63.5 \\
    M4 & 4.6 & \textbf{49.4} & \textbf{16.7} & \textbf{38.8} & \textbf{23.6} & 54.5 & \textbf{22.9} & \textbf{52.7} & \textbf{8.6} & \textbf{70.6} & \textbf{33.3} & \textbf{64.0} \\
    \hline
  \end{tabular}
  \caption{Performance comparison of module combination configurations. M4 (fully aligned) achieves best performance overall.}
  \label{tab:module-combination}
\end{table*}

\paragraph{Additional Experiment: Qwen3-0.6B LLM.}
We further experimented with Qwen3-0.6B as the LLM backbone (M5) to understand the effect of different small LLM choices.
Table~\ref{tab:qwen3-llm} compares M5 with M1 (Qwen2.5-0.5B) and M2 (Qwen2.5-3B). 
All models here use the same audio encoder as M1.

We can see that M5 performs significantly worse than M1 despite similar model sizes (both $\sim$1.1B) on particular tasks (e.g., Clotho A2T).
We do not have a good explanation for this yet.
My guess is that the Qwen3-Omni LLM differs more significantly from the Qwen3 text-only LLM, which results in the performance gap.
Specifically, the Qwen3-Omni LLM is implemented with the type \textit{Qwen3OmniMoePreTrainedModel}, while Qwen3-0.6B is implemented with the type \textit{Qwen3ForCausalLM}.
To ensure our code works, I randomly initialized an Omni model with an LLM shape consistent with Qwen3-0.6B, and then copied the parameters of the Qwen3-0.6B model into the corresponding locations.
This simple copying approach allows the forward and backward passes to work, but might not unlock the full potential of the LLM for audio tasks.

\begin{table*}[t]
  \centering
  \scriptsize
  \begin{tabular}{lcccccccccccc}
    \hline
    \textbf{Model} & \multicolumn{2}{c}{\textbf{Clotho A2T}} & \multicolumn{2}{c}{\textbf{Clotho T2A}} & \multicolumn{2}{c}{\textbf{AudioSetStrong A2T}} & \multicolumn{2}{c}{\textbf{AudioSetStrong T2A}} & \multicolumn{2}{c}{\textbf{AudioCaps A2T}} & \multicolumn{2}{c}{\textbf{AudioCaps T2A}} \\
    & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 & R@1 & cvR@5 \\
    \hline
    M1 & 4.0 & 42.7 & 10.6 & 28.8 & 18.8 & 47.1 & 15.6 & 40.2 & 7.2 & 61.8 & 22.2 & 49.7 \\
    M2 & \textbf{4.1} & \textbf{46.3} & \textbf{13.7} & \textbf{35.1} & \textbf{21.5} & \textbf{50.2} & \textbf{19.5} & \textbf{50.8} & \textbf{7.5} & \textbf{66.8} & \textbf{29.0} & \textbf{59.3} \\
    M5 & 2.3 & 31.9 & 9.3 & 25.4 & 13.5 & 36.5 & 10.2 & 33.2 & 5.5 & 54.3 & 20.4 & 48.5 \\
    \hline
  \end{tabular}
  \caption{Comparison of different small LLM backbones. M2 (Qwen2.5-3B) performs best overall. M5 (Qwen3-0.6B) performs worse than M1 despite similar sizes.}
  \label{tab:qwen3-llm}
\end{table*}

\bibliography{custom}
\end{document}