arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.04552cs.CLcs.SDeess.AS
收藏

用于长尾语音识别的查找表循环语言模型

Lookup-Table Recurrent Language Models for Long Tail Speech Recognition

Google Research(谷歌研究院)

W. Ronny Huang, Tara N. Sainath, Cal Peyser, Shankar Kumar, David Rybach, Trevor Strohman

摘要:

我们引入了查找表语言模型(LookupLM),这是一种通过增加嵌入表的表达能力来扩大RNN语言模型大小的方法,只需不断增加浮点运算。特别是,我们实例化了一个(额外的)嵌入表,它嵌入了之前的n-gram令牌序列,而不是单个令牌。这使得嵌入表可以任意地扩展--性能也相应地提高--而不需要改变标记词汇表。由于嵌入是通过查找从表中稀疏地检索的;增加表的大小既不会给每个前向传递增加额外的操作,也不会增加需要存储在有限的GPU/TPU内存上的额外参数。我们探索将n-gram嵌入表扩展到近十亿个参数。当在30亿句语料库上训练时,我们发现LookupLM在标准RNN语言模型基线上将下游语音识别任务的长尾对数困惑度提高了2.44,长尾WER提高了23.4%,这一改进与将基线放大6.2倍浮点运算的数量相当。

英文摘要:

We introduce Lookup-Table Language Models (LookupLM), a method for scaling up the size of RNN language models with only a constant increase in the floating point operations, by increasing the expressivity of the embedding table. In particular, we instantiate an (additional) embedding table which embeds the previous n-gram token sequence, rather than a single token. This allows the embedding table to be scaled up arbitrarily -- with a commensurate increase in performance -- without changing the token vocabulary. Since embeddings are sparsely retrieved from the table via a lookup; increasing the size of the table adds neither extra operations to each forward pass nor extra parameters that need to be stored on limited GPU/TPU memory. We explore scaling n-gram embedding tables up to nearly a billion parameters. When trained on a 3-billion sentence corpus, we find that LookupLM improves long tail log perplexity by 2.44 and long tail WER by 23.4% on a downstream speech recognition task over a standard RNN language model baseline, an improvement comparable to a scaling up the baseline by 6.2x the number of floating point operations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑