多模态上下文学习用于低资源语言的语音识别
Multimodal In-context Learning for ASR of Low-resource Languages
AI总结：
本文探讨了通过多模态上下文学习提升低资源语言语音识别性能的方法，分析了跨语言迁移学习对模型效率的影响，并提出结合更强声学模型与语音LLM的改进系统。
摘要：
自动语音识别（ASR）仍只覆盖了世界语言的一小部分，主要由于监督数据稀缺。上下文学习（ICL）结合大型语言模型（LLMs）解决了这一问题，但先前研究主要集中在训练覆盖的高资源语言和纯文本设置。本文探讨了语音LLMs是否能通过多模态上下文学习（MICL）学习未见过的语言，并如何利用此学习提升ASR。我们使用Phi-4和Qwen3-Omni两个语音LLMs，在三种多样化的濒危语言上进行实验。首先发现MICL对未见过的语言有效，利用语音和文本模态。我们进一步表明，跨语言迁移学习能提高MICL在目标语言上的效率，无需训练。此外，我们分析了注意力模式以解释MICL机制，并观察到音频和文本上下文之间存在层依赖性的偏好，整体偏向文本。最后，我们发现基于提示的ASR在未见过的语言上表现不佳，促使提出一个简单的ASR系统，结合更强的声学模型与语音LLM，通过基于MICL的声学假设选择。结果表明，MICL一致提升了ASR性能，跨语言迁移学习在不使用目标语言数据的情况下可匹配或优于基于语料库训练的语言模型。我们的代码已公开可用。
英文摘要：
Automatic speech recognition (ASR) still covers only a small fraction of the world's languages, mainly due to supervised data scarcity. In-context learning (ICL) with large language models (LLMs) addresses this problem, but prior work largely focuses on high-resource languages covered during training and text-only settings. This paper investigates whether speech LLMs can learn unseen languages with multimodal ICL (MICL), and how this learning can be used to improve ASR. We conduct experiments with two speech LLMs, Phi-4 and Qwen3-Omni, on three diverse endangered languages. Firstly, we find that MICL is effective for unseen languages, leveraging both speech and text modalities. We further show that cross-lingual transfer learning improves MICL efficiency on target languages without training on them. Moreover, we analyze attention patterns to interpret MICL mechanisms, and we observe layer-dependent preferences between audio and text context, with an overall bias towards text. Finally, we show that prompt-based ASR with speech LLMs performs poorly on unseen languages, motivating a simple ASR system that combines a stronger acoustic model with a speech LLM via MICL-based selection of acoustic hypotheses. Results show that MICL consistently improves ASR performance, and that cross-lingual transfer learning matches or outperforms corpus-trained language models without using target-language data. Our code is publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
