MoWE-Audio：基于弱编码器混合的多任务音频大语言模型
MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders
AI总结：
本文提出MoWE-Audio，通过混合弱编码器增强AudioLLM的特征提取能力，在不显著增加模型规模的情况下提升多任务性能，拓宽音频任务适用性。
摘要：
大语言模型（LLMs）的快速发展显著增强了自然语言处理能力，推动了能够处理语音和音频输入以及文本的AudioLLMs的发展。现有的AudioLLMs通常将预训练的音频编码器与预训练的LLM相结合，随后在特定音频任务上进行微调。然而，预训练的音频编码器在捕获新任务和新数据集的特征方面能力有限。为了解决这一问题，我们提出将“弱”编码器的混合（MoWE）引入AudioLLM框架。MoWE通过一个相对轻量级的编码器池补充基础编码器，根据音频输入选择性地激活这些编码器，以在不显著增加模型规模的情况下增强特征提取。我们的实验结果表明，MoWE有效提升了多任务性能，拓宽了AudioLLMs在更多样化音频任务中的适用性。
英文摘要：
The rapid advancements in large language models (LLMs) have significantly enhanced natural language processing capabilities, facilitating the development of AudioLLMs that process and understand speech and audio inputs alongside text. Existing AudioLLMs typically combine a pre-trained audio encoder with a pre-trained LLM, which are subsequently finetuned on specific audio tasks. However, the pre-trained audio encoder has constrained capacity to capture features for new tasks and datasets. To address this, we propose to incorporate mixtures of `weak' encoders (MoWE) into the AudioLLM framework. MoWE supplements a base encoder with a pool of relatively light weight encoders, selectively activated based on the audio input to enhance feature extraction without significantly increasing model size. Our empirical results demonstrate that MoWE effectively improves multi-task performance, broadening the applicability of AudioLLMs to more diverse audio tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
