用于多语言对话式自动语音识别的双向上下文增强语音大语言模型
Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR
AI总结：
本文提出将双向上下文融入语音大语言模型，通过字符级上下文掩码训练和两阶段上下文感知解码，在1500小时多语言对话语料上取得18%的相对改进，优于6000小时数据训练的模型。
摘要：
本文介绍了将语言特定的双向上下文集成到语音大语言模型（SLLM）中，以改进多语言连续对话式自动语音识别（ASR）。我们提出了一种训练期间的字符级上下文掩码策略，该策略随机移除部分上下文，以增强鲁棒性并更好地模拟推理过程中可能出现的错误转录。在解码方面，采用两阶段流程：首先进行孤立的片段解码，然后利用相邻假设进行上下文感知的重新解码。在涵盖十一种语言的1500小时多语言对话语音与语言模型（MLC-SLM）语料库上进行评估，与强基线相比，我们的方法取得了18%的相对改进，甚至优于在MLC-SLM竞赛中使用6000小时数据训练的模型。这些结果凸显了在多语言连续对话式ASR中融入上下文信息的显著益处。
英文摘要：
This paper introduces the integration of language-specific bi-directional context into a speech large language model (SLLM) to improve multilingual continuous conversational automatic speech recognition (ASR). We propose a character-level contextual masking strategy during training, which randomly removes portions of the context to enhance robustness and better emulate the flawed transcriptions that may occur during inference. For decoding, a two-stage pipeline is utilized: initial isolated segment decoding followed by context-aware re-decoding using neighboring hypotheses. Evaluated on the 1500-hour Multilingual Conversational Speech and Language Model (MLC-SLM) corpus covering eleven languages, our method achieves an 18% relative improvement compared to a strong baseline, outperforming even the model trained on 6000 hours of data for the MLC-SLM competition. These results underscore the significant benefit of incorporating contextual information in multilingual continuous conversational ASR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
