arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.15045eess.AS
收藏

大语言模型与语音:整合与结合

LLMs and Speech: Integration vs. Combination

RWTH Aachen University(亚琛工业大学);AppTek GmbH(AppTek有限公司)

Robin Schmitt, Albert Zeyer, Mohammad Zeineldeen, Ralf Schlüter, Hermann Ney

AI总结:

本文研究如何有效利用预训练大语言模型进行自动语音识别,比较了将语音模型与大语言模型紧密整合与传统浅层融合方法的优劣,探讨了多种优化策略及联合识别技术。

摘要:

在本工作中,我们研究如何最有效地利用预训练的大语言模型(LLM)进行自动语音识别。具体而言,我们比较了将语音模型(AM)与LLM('语音LLM')紧密整合与传统通过浅层融合将AM和LLM结合的方法。对于紧密整合,我们对不同标签单元、微调策略、LLM大小和预训练数据、注意力接口、编码器下采样、文本提示和长度归一化的效果进行了消融分析。此外,我们研究了与CTC模型联合识别以缓解语音LLM的幻觉现象,并提出了有效的优化方法。对于浅层融合,我们探讨了使用不同标签单元微调LLM对转录结果的影响,并比较了对AM假设进行重评分与单次识别在标签层面或延迟融合AM和LLM得分时的效果。我们使用LibriSpeech和Loquacious进行训练,并在HuggingFace ASR排行榜上评估我们的模型。

英文摘要:

In this work, we study different approaches to utilize large language models (LLMs) for automatic speech recognition (ASR). Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion and provide ablations on the effect of different label units and LLM sizes. For tight integration, we further examine the effect of attention interfaces, encoder downsampling, and length normalization. Furthermore, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations. We train and evaluate on LibriSpeech and Loquacious and additionally evaluate on the HuggingFace ASR leaderboard. Across model sizes, we find that shallow fusion consistently outperforms tight integration of AM and LLM on in-domain data, highlighting the importance of strong shallow-fusion baselines when evaluating speech LLMs for ASR. On the more heterogeneous HuggingFace ASR leaderboard, however, the integrated prefix LLM achieves lower average WER than shallow fusion, with gains concentrated on out-of-domain corpora.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑