arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.08583eess.AScs.AIcs.CLcs.LGcs.SD
收藏

JEIT:用于语音识别的联合端到端模型和内部语言模型训练

JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognition

Google LLC.(谷歌有限责任公司)

Zhong Meng, Weiran Wang, Rohit Prabhavalkar, Tara N. Sainath, Tongzhou Chen, Ehsan Variani, Yu Zhang, Bo Li, Andrew Rosenberg, Bhuvana Ramabhadran

摘要:

本文提出了一种端到端(E2 E)模型和内部语言模型(ILM)联合训练方法JEIT,在E2 E训练过程中将大规模不成对文本注入ILM,提高了稀有词语音识别的性能。使用JEIT,E2 E模型计算音频-转录本对的E2 E损失,而其ILM估计未配对文本的交叉熵损失。E2 E模型被训练以最小化E2 E和ILM损失的加权和。在JEIT过程中,ILM从不成对的文本中吸收知识,而E2 E训练作为正则化。与ILM自适应方法不同,JEIT不需要单独的自适应步骤,并且避免了对ILM的Kullback-Leibler发散正则化的需要。我们还表明,模块化混合自回归传感器(MHAT)的JEIT框架中的性能优于HAT,并且在ILM适应过程中比HAT更鲁棒。为了突破不成对文本注入的限制,我们进一步提出了一种组合JEIT和JOIST训练(CJJT),它受益于模态匹配,编码器文本注入和ILM训练。JEIT和CJJT都可以促进更有效的LM融合。使用100 B未配对句子,JEIT/CJJT将罕见词识别准确率提高了16.4%,而不是在没有未配对文本的情况下训练的模型。

英文摘要:

We propose JEIT, a joint end-to-end (E2E) model and internal language model (ILM) training method to inject large-scale unpaired text into ILM during E2E training which improves rare-word speech recognition. With JEIT, the E2E model computes an E2E loss on audio-transcript pairs while its ILM estimates a cross-entropy loss on unpaired text. The E2E model is trained to minimize a weighted sum of E2E and ILM losses. During JEIT, ILM absorbs knowledge from unpaired text while the E2E training serves as regularization. Unlike ILM adaptation methods, JEIT does not require a separate adaptation step and avoids the need for Kullback-Leibler divergence regularization of ILM. We also show that modular hybrid autoregressive transducer (MHAT) performs better than HAT in the JEIT framework, and is much more robust than HAT during ILM adaptation. To push the limit of unpaired text injection, we further propose a combined JEIT and JOIST training (CJJT) that benefits from modality matching, encoder text injection and ILM training. Both JEIT and CJJT can foster a more effective LM fusion. With 100B unpaired sentences, JEIT/CJJT improves rare-word recognition accuracy by up to 16.4% over a model trained without unpaired text.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑