面向多语言指令遵循语音大语言模型的语言感知蒸馏：仅使用ASR监督
Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision
AI总结：
本文提出一种语言感知蒸馏方法，通过查询库和门控网络提升多语言指令遵循语音大语言模型的性能，实现在Audio-MLQA基准上比现有基线提升32%。
摘要：
语音大语言模型（LLMs）能够理解和遵循多种语言的指令，在现实世界交互中非常有用，但使用监督微调训练这些模型却困难重重，需要大量的任务特定语音语料库。尽管最近基于蒸馏的方法通过仅使用标注的ASR数据，并通过轻量级投影器对文本和语音进行对齐，来训练高性能的英语语音LLMs，但这些模型在扩展到多语言设置时表现不佳，因为共享投影器中的语言干扰。我们通过引入语言感知蒸馏，使用查询库和门控网络，通过Q-Former投影器选择或混合查询标记来解决这一问题。我们的方法在指令遵循任务上比匹配的多语言蒸馏基线提高了14%。我们进一步合成Audio-MLQA，一个基于MLQA的多语言语音问答基准，包含高质量的TTS问题。我们的最佳模型在Audio-MLQA上比现有的语音LLM基线提高了32%。
英文摘要：
Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
