利用大语言模型提升多说话人自动语音识别性能
Advancing Multi-talker ASR Performance with Large Language Models
AI总结：
本文提出基于大语言模型的序列化输出训练方法用于多说话人ASR，通过微调预训练语音编码器和LLM，在LibriMix和AMI数据集上超越传统AED方法并取得最先进性能。
摘要：
在对话场景中识别多个说话人的重叠语音是自动语音识别（ASR）最具挑战性的问题之一。序列化输出训练（SOT）是解决多说话人ASR的经典方法，其思想是根据多个说话人语音的发出时间将他们的转录文本串联起来进行训练。然而，SOT风格的转录文本源自对话中多个相关话语的串联，在很大程度上依赖于对长上下文的建模。因此，与在基于注意力的编码器-解码器（AED）架构中主要强调编码器性能的传统方法相比，一种利用大语言模型（LLM）并发挥预训练解码器能力的新方法可能更适合这种复杂且具有挑战性的场景。在本文中，我们提出了一种基于LLM的SOT方法用于多说话人ASR，利用预训练的语音编码器和LLM，并使用适当的策略在多说话人数据集上对它们进行微调。实验结果表明，我们的方法在模拟数据集LibriMix上超越了传统的基于AED的方法，并在真实世界数据集AMI的评估集上取得了最先进的性能，优于先前工作中使用多1000倍监督数据训练的AED模型。
英文摘要：
Recognizing overlapping speech from multiple speakers in conversational scenarios is one of the most challenging problem for automatic speech recognition (ASR). Serialized output training (SOT) is a classic method to address multi-talker ASR, with the idea of concatenating transcriptions from multiple speakers according to the emission times of their speech for training. However, SOT-style transcriptions, derived from concatenating multiple related utterances in a conversation, depend significantly on modeling long contexts. Therefore, compared to traditional methods that primarily emphasize encoder performance in attention-based encoder-decoder (AED) architectures, a novel approach utilizing large language models (LLMs) that leverages the capabilities of pre-trained decoders may be better suited for such complex and challenging scenarios. In this paper, we propose an LLM-based SOT approach for multi-talker ASR, leveraging pre-trained speech encoder and LLM, fine-tuning them on multi-talker dataset using appropriate strategies. Experimental results demonstrate that our approach surpasses traditional AED-based methods on the simulated dataset LibriMix and achieves state-of-the-art performance on the evaluation set of the real-world dataset AMI, outperforming the AED model trained with 1000 times more supervised data in previous works.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
