Transducer-Llama：将LLM集成到可流式处理的Transducer语音识别中
Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition
AI总结：
本文提出Transducer-Llama架构，通过将LLM集成到Factorized Transducer模型实现流式ASR，并采用弱到强LM替换与MWER微调策略，在LibriSpeech上较基线最高降低32%词错率。
摘要：
尽管大型语言模型（LLM）已被应用于自动语音识别（ASR），但使模型可流式处理仍是一项挑战。本文提出了一种新颖的模型架构Transducer-Llama，将LLM集成到Factorized Transducer（FT）模型中，自然地实现了流式处理能力。此外，鉴于LLM的大词汇表会导致数据稀疏问题并增加口语系统的训练成本，本文引入了一种高效的词汇适应技术，以将LLM与语音系统词汇表对齐。结果表明，使用基于强预训练LLM的预测器通过RNN-T损失直接优化FT模型，比较小的预训练LM预测器仅带来有限改善。因此，本文提出弱到强LM替换策略，在RNN-T损失训练期间使用弱LM预测器，随后将其替换为强LLM。LM替换后，采用最小词错误率（MWER）损失微调LLM预测器与Transducer-Llama模型的集成。在LibriSpeech和大规模多语言LibriSpeech语料库上的实验表明，所提出的流式Transducer-Llama方法较强FT基线实现了17%的相对词错率降低（WERR），较RNN-T基线实现了32%的WERR。
英文摘要：
While large language models (LLMs) have been applied to automatic speech recognition (ASR), the task of making the model streamable remains a challenge. This paper proposes a novel model architecture, Transducer-Llama, that integrates LLMs into a Factorized Transducer (FT) model, naturally enabling streaming capabilities. Furthermore, given that the large vocabulary of LLMs can cause data sparsity issue and increased training costs for spoken language systems, this paper introduces an efficient vocabulary adaptation technique to align LLMs with speech system vocabularies. The results show that directly optimizing the FT model with a strong pre-trained LLM-based predictor using the RNN-T loss yields some but limited improvements over a smaller pre-trained LM predictor. Therefore, this paper proposes a weak-to-strong LM swap strategy, using a weak LM predictor during RNN-T loss training and then replacing it with a strong LLM. After LM replacement, the minimum word error rate (MWER) loss is employed to finetune the integration of the LLM predictor with the Transducer-Llama model. Experiments on the LibriSpeech and large-scale multi-lingual LibriSpeech corpora show that the proposed streaming Transducer-Llama approach gave a 17% relative WER reduction (WERR) over a strong FT baseline and a 32% WERR over an RNN-T baseline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
