Softmax与BPE和Hybrid-LightRNN的快速简单混合用于语言生成
Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation
摘要：
Softmax混合模型（MoS）已被证明可以有效地解决基于Softmax模型的表达能力限制。尽管有已知的优势，但由于需要计算多个Softmax，MoS实际上被其大量的内存消耗和计算时间所密封。在这项工作中，我们开始释放的MoS在实际应用中的力量，通过调查改进的单词编码方案，这可以有效地减少词汇量，从而减轻内存和计算负担。我们证明了BPE和我们提出的Hybrid-LightRNN都可以改进编码机制，可以将MoS的时间和内存消耗减半，而不会造成性能损失。使用MoS，我们在IWITH2014德语到英语语料库上实现了1.5 BLEU分数的改进，在图像描述上实现了0.76 CIDER分数的改进。此外，在更大的WMT 2014机器翻译数据集上，我们的MoS增强型Transformer在英语到德语的BLEU得分为29.5，在英语到法语的BLEU得分为42.1，分别比单一Softmax Transformer高出0.8和0.4，在WMT 2014英语到德语任务上达到了最先进的结果。
英文摘要：
Mixture of Softmaxes (MoS) has been shown to be effective at addressing the expressiveness limitation of Softmax-based models. Despite the known advantage, MoS is practically sealed by its large consumption of memory and computational time due to the need of computing multiple Softmaxes. In this work, we set out to unleash the power of MoS in practical applications by investigating improved word coding schemes, which could effectively reduce the vocabulary size and hence relieve the memory and computation burden. We show both BPE and our proposed Hybrid-LightRNN lead to improved encoding mechanisms that can halve the time and memory consumption of MoS without performance losses. With MoS, we achieve an improvement of 1.5 BLEU scores on IWSLT 2014 German-to-English corpus and an improvement of 0.76 CIDEr score on image captioning. Moreover, on the larger WMT 2014 machine translation dataset, our MoS-boosted Transformer yields 29.5 BLEU score for English-to-German and 42.1 BLEU score for English-to-French, outperforming the single-Softmax Transformer by 0.8 and 0.4 BLEU scores respectively and achieving the state-of-the-art result on WMT 2014 English-to-German task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
