从教师那里学习拼写：将知识从语言模型转移到序列到序列的语音识别
Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition
摘要：
将外部语言模型集成到序列到序列语音识别系统中是不平凡的。以前的作品利用线性插值或融合网络来集成外部语言模型。然而，这些方法引入了外部组件，并增加了解码计算量。在本文中，我们提出了一种基于知识蒸馏的训练方法，将外部语言模型集成到序列到序列模型中。在大规模外部文本上训练的递归神经网络语言模型生成软标签来指导序列到序列模型训练。因此，语言模型扮演着教师的角色。这种方法在测试过程中不会向序列到序列模型添加任何外部组件。该方法可以灵活地与浅层融合技术结合进行解码。实验在中文公共数据集AISHELL-1和CLMAD上进行。我们的方法实现了一个字符的错误率为9.3%，这是相对减少了18.42%相比，香草序列到序列模型。
英文摘要：
Integrating an external language model into a sequence-to-sequence speech recognition system is non-trivial. Previous works utilize linear interpolation or a fusion network to integrate external language models. However, these approaches introduce external components, and increase decoding computation. In this paper, we instead propose a knowledge distillation based training approach to integrating external language models into a sequence-to-sequence model. A recurrent neural network language model, which is trained on large scale external text, generates soft labels to guide the sequence-to-sequence model training. Thus, the language model plays the role of the teacher. This approach does not add any external component to the sequence-to-sequence model during testing. And this approach is flexible to be combined with shallow fusion technique together for decoding. The experiments are conducted on public Chinese datasets AISHELL-1 and CLMAD. Our approach achieves a character error rate of 9.3%, which is relatively reduced by 18.42% compared with the vanilla sequence-to-sequence model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
