arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.10446cs.CLcs.AIcs.LGcs.SDeess.AS
收藏

大型语言模型是噪声鲁棒语音识别的高效学习器

Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

Nanyang Technological University(南洋理工大学);Georgia Institute of Technology(佐治亚理工学院);NVIDIA Research(英伟达研究院);University of Aberdeen(阿伯丁大学);Tsinghua University(清华大学)

Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, EnSiong Chng

AI总结:

本文提出从N-best列表提取语言空间噪声嵌入并结合互信息知识蒸馏,使LLMs在有限数据下实现噪声鲁棒ASR生成式纠错,词错误率最高改善53.9%。

摘要:

大型语言模型(LLMs)的最新进展推动了用于自动语音识别(ASR)的生成式纠错(GER)的发展,该方法利用LLMs丰富的语言知识和强大的推理能力来改进识别结果。最新的工作提出了一个基于HyPoradise数据集的GER基准,通过高效的LLM微调来学习从ASR N-best假设到真实转录的映射,该工作展示了显著的有效性,但在噪声鲁棒ASR方面缺乏针对性。在本工作中,我们将该基准扩展到噪声条件,并研究是否能够教会LLMs像鲁棒ASR那样为GER执行去噪,其中一种解决方案是将噪声信息作为条件器引入LLM。然而,直接融入来自音频编码器的噪声嵌入可能由于跨模态差距而损害LLM的微调。为此,我们提出从N-best列表中提取一个语言空间噪声嵌入来表示源语音的噪声条件,这可以促进GER中的去噪过程。此外,为了增强其对音频噪声的表示能力,我们设计了一种基于互信息估计的知识蒸馏(KD)方法,将音频嵌入中的真实噪声信息蒸馏到我们的语言嵌入中。在多种最新LLMs上的实验表明,我们的方法在词错误率方面取得了新的突破,在训练数据有限的情况下实现了高达53.9%的纠错改进。分析表明,我们的语言空间噪声嵌入能够很好地表示源语音的噪声条件,在此条件下,现成的LLMs展现出强大的语言空间去噪能力。

英文摘要:

Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which leverages the rich linguistic knowledge and powerful reasoning ability of LLMs to improve recognition results. The latest work proposes a GER benchmark with HyPoradise dataset to learn the mapping from ASR N-best hypotheses to ground-truth transcription by efficient LLM finetuning, which shows great effectiveness but lacks specificity on noise-robust ASR. In this work, we extend the benchmark to noisy conditions and investigate if we can teach LLMs to perform denoising for GER just like what robust ASR do}, where one solution is introducing noise information as a conditioner into LLM. However, directly incorporating noise embeddings from audio encoder could harm the LLM tuning due to cross-modality gap. To this end, we propose to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech, which can promote the denoising process in GER. Furthermore, in order to enhance its representation ability of audio noise, we design a knowledge distillation (KD) approach via mutual information estimation to distill the real noise information in audio embeddings to our language embedding. Experiments on various latest LLMs demonstrate our approach achieves a new breakthrough with up to 53.9% correction improvement in terms of word error rate while with limited training data. Analysis shows that our language-space noise embedding can well represent the noise conditions of source speech, under which off-the-shelf LLMs show strong ability of language-space denoising.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑