通过多指令训练教会多语言大语言模型理解多语言语音
Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training
AI总结：
针对大语言模型难以应用于语音领域的问题，提出融合多语言大语言模型与语音编码器的BLOOMZMMS模型，通过多指令训练实现知识跨模态迁移，在139种语言的多任务零样本评估中表现鲁棒。
摘要：
语言建模的最新进展催生了能够执行各类自然语言处理任务的大语言模型（Large Language Models, LLMs）。尽管大语言模型在文本类任务中表现出色，但将其应用于语音领域仍存在局限性且面临挑战。本文提出BLOOMZMMS这一新型模型，它将多语言大语言模型与多语言语音编码器相结合，旨在发挥大语言模型的能力以应用于语音识别及更多相关任务。我们采用多指令训练方法，验证了语言知识从文本模态到语音模态的可迁移性。我们基于139种语言、总时长1900小时的转写数据开展实验，结果表明模型能够有效学习多语言语音表示，并将其与多语言大语言模型对齐。虽然初始学习到的表示在任务泛化性上存在局限，但我们通过生成多指令风格的合成目标解决了这一问题。我们的零样本评估结果证实，该方法在语音翻译、多语言口语语言理解等多项任务中均具备鲁棒性，从而为大语言模型在语音领域的应用开辟了新路径。
英文摘要：
Recent advancements in language modeling have led to the emergence of Large Language Models (LLMs) capable of various natural language processing tasks. Despite their success in text-based tasks, applying LLMs to the speech domain remains limited and challenging. This paper presents BLOOMZMMS, a novel model that integrates a multilingual LLM with a multilingual speech encoder, aiming to harness the capabilities of LLMs for speech recognition and beyond. Utilizing a multi-instructional training approach, we demonstrate the transferability of linguistic knowledge from the text to the speech modality. Our experiments, conducted on 1900 hours of transcribed data from 139 languages, establish that a multilingual speech representation can be effectively learned and aligned with a multilingual LLM. While this learned representation initially shows limitations in task generalization, we address this issue by generating synthetic targets in a multi-instructional style. Our zero-shot evaluation results confirm the robustness of our approach across multiple tasks, including speech translation and multilingual spoken language understanding, thereby opening new avenues for applying LLMs in the speech domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
