通过学习音频-文本跨模态上下文表示的对话语音识别
Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation
AI总结：
本文提出一种基于跨模态对话表示的对话ASR系统，通过结合预训练语音文本模型与条件潜变量模块提取历史上下文与对话级属性，在HKUST和MagicData-RAMC数据集上相对标准Conformer分别提升8.8%和23%准确率。
摘要：
对话场景中的自动语音识别（ASR）面临独特挑战，包括从先前的对话轮次中提取相关上下文信息。由于无关内容、错误传播和冗余，现有方法难以提取更长且更有效的上下文。为解决此问题，我们引入了一种新颖的对话ASR系统，通过跨模态对话表示扩展了Conformer编码器-解码器模型。该方法利用跨模态提取器，通过专用编码器和模态级掩码输入结合预训练的语音和文本模型。这能够在没有显式错误传播的情况下提取更丰富的历史语音上下文。我们还引入了条件潜变量模块，以学习对话级属性，如角色偏好和话题连贯性。通过将跨模态和对话表示引入解码器，我们的模型在较长句子中保留上下文且无信息丢失，与标准Conformer模型相比，在中文对话数据集HKUST和MagicData-RAMC上分别实现了8.8%和23%的相对准确率提升。
英文摘要：
Automatic Speech Recognition (ASR) in conversational settings presents unique challenges, including extracting relevant contextual information from previous conversational turns. Due to irrelevant content, error propagation, and redundancy, existing methods struggle to extract longer and more effective contexts. To address this issue, we introduce a novel conversational ASR system, extending the Conformer encoder-decoder model with cross-modal conversational representation. Our approach leverages a cross-modal extractor that combines pre-trained speech and text models through a specialized encoder and a modal-level mask input. This enables the extraction of richer historical speech context without explicit error propagation. We also incorporate conditional latent variational modules to learn conversational level attributes such as role preference and topic coherence. By introducing both cross-modal and conversational representations into the decoder, our model retains context over longer sentences without information loss, achieving relative accuracy improvements of 8.8% and 23% on Mandarin conversation datasets HKUST and MagicData-RAMC, respectively, compared to the standard Conformer model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
