延迟融合：将大语言模型集成到端到端语音识别的首遍解码中
Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition
AI总结：
针对端到端语音识别集成大语言模型时的高计算成本与词表不匹配问题，本文提出延迟融合方法，通过延迟应用LLM分数减少推理调用与打分假设数，支持重分词，实验证实其在解码速度和准确率上均优于浅融合与N-best重打分。
摘要：
本文提出了一种结合大语言模型（LLM）的端到端自动语音识别（E2E-ASR）高效解码方法。尽管浅融合是将语言模型集成到E2E-ASR解码中的最常用方法，但在使用LLM时我们面临两个实际问题：（1）LLM推理计算成本高昂；（2）ASR模型与LLM之间可能存在词表不匹配问题。要解决这种不匹配，我们需要重新训练ASR模型和/或LLM，这充其量是耗时的，且在很多情况下并不可行。我们提出“延迟融合（delayed fusion）”方法，该方法在解码过程中延迟将LLM分数应用于ASR假设，使得预训练LLM能更便捷地应用于ASR任务。该方法不仅能减少需要LLM打分的假设数量，还能减少LLM推理调用的次数；如果ASR和LLM采用不同的分词方式，它还支持在解码过程中对ASR假设进行重新分词。我们使用LibriHeavy ASR语料库以及三个公开LLM（OpenLLaMA 3B、7B和Mistral 7B）进行实验，结果表明，与浅融合和N-best重打分相比，延迟融合提升了解码速度和准确率。
英文摘要：
This paper presents an efficient decoding approach for end-to-end automatic speech recognition (E2E-ASR) with large language models (LLMs). Although shallow fusion is the most common approach to incorporate language models into E2E-ASR decoding, we face two practical problems with LLMs. (1) LLM inference is computationally costly. (2) There may be a vocabulary mismatch between the ASR model and the LLM. To resolve this mismatch, we need to retrain the ASR model and/or the LLM, which is at best time-consuming and in many cases not feasible. We propose "delayed fusion," which applies LLM scores to ASR hypotheses with a delay during decoding and enables easier use of pre-trained LLMs in ASR tasks. This method can reduce not only the number of hypotheses scored by the LLM but also the number of LLM inference calls. It also allows re-tokenizion of ASR hypotheses during decoding if ASR and LLM employ different tokenizations. We demonstrate that delayed fusion provides improved decoding speed and accuracy compared to shallow fusion and N-best rescoring using the LibriHeavy ASR corpus and three public LLMs, OpenLLaMA 3B & 7B and Mistral 7B.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
