增强语音大语言模型泛化能力的多任务行为模仿与语音-文本交错方法
Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving
AI总结：
针对语音大语言模型因缺乏标注数据导致泛化差的问题，提出仅依赖配对语音与文本的多任务行为模仿方法MTBI，结合语音-文本交错增强对齐效率，在提示与任务泛化上超越现有最优模型且所需监督数据更少。
摘要：
大语言模型（LLMs）在跨任务泛化方面展现出显著能力，这引发了将语音与LLMs集成的日益关注。这些语音大语言模型（SLLMs）通常采用监督微调来将语音与基于文本的LLMs对齐。然而，在广泛任务范围内缺乏带注释的语音数据，阻碍了对齐效率，导致泛化能力较差。为解决这些问题，我们提出了一种新颖的多任务“行为模仿”方法，并结合语音-文本交错技术，称为MTBI，该方法仅依赖于配对的语音和转录文本。通过确保LLM解码器对配对的语音和文本生成等效响应，我们实现了更具泛化能力的SLLM。交错技术用于进一步增强对齐效率。我们引入了一个简单的基准测试，以评估不同模型在提示泛化和任务泛化方面的表现。实验结果表明，我们的MTBI在提示泛化和任务泛化方面均优于最先进的SLLMs，同时所需的监督语音数据更少。
英文摘要：
Large language models (LLMs) have shown remarkable generalization across tasks, leading to increased interest in integrating speech with LLMs. These speech LLMs (SLLMs) typically use supervised fine-tuning to align speech with text-based LLMs. However, the lack of annotated speech data across a wide range of tasks hinders alignment efficiency, resulting in poor generalization. To address these issues, we propose a novel multi-task 'behavior imitation' method with speech-text interleaving, called MTBI, which relies solely on paired speech and transcripts. By ensuring the LLM decoder generates equivalent responses to paired speech and text, we achieve a more generalized SLLM. Interleaving is used to further enhance alignment efficiency. We introduce a simple benchmark to evaluate prompt and task generalization across different models. Experimental results demonstrate that our MTBI outperforms SOTA SLLMs on both prompt and task generalization, while requiring less supervised speech data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
