用深度级 LSTM 重构 Transformer 连接
Rewiring the Transformer with Depth-Wise LSTMs
AI总结：
本文提出用深度级 LSTM 连接并聚合 Transformer 层与子层，以替代残差路径的信息传递，显著提升机器翻译 BLEU 并改善深层模型收敛。
摘要：
堆叠非线性层使深度神经网络能够建模复杂函数，而在 Transformer 层中加入残差连接有利于收敛和性能。然而，残差连接可能使模型“遗忘”距离较远的层，无法有效融合来自先前层的信息。选择性地管理 Transformer 层的表示聚合可能带来更好的性能。本文提出一种带有深度级 LSTM 的 Transformer，用其连接级联的 Transformer 层和子层。我们表明，Transformer 层内的层归一化和前馈计算可以被吸收进连接纯 Transformer 注意力层的深度级 LSTM 中。基于 6 层 Transformer 的实验显示，在 WMT 14 英语—德语/法语任务以及 OPUS-100 多对多多语言 NMT 任务中，BLEU 均有显著提升；深度 Transformer 实验也证明了深度级 LSTM 对深度 Transformer 收敛和性能的有效性。
英文摘要：
Stacking non-linear layers allows deep neural networks to model complicated functions, and including residual connections in Transformer layers is beneficial for convergence and performance. However, residual connections may make the model "forget" distant layers and fail to fuse information from previous layers effectively. Selectively managing the representation aggregation of Transformer layers may lead to better performance. In this paper, we present a Transformer with depth-wise LSTMs connecting cascading Transformer layers and sub-layers. We show that layer normalization and feed-forward computation within a Transformer layer can be absorbed into depth-wise LSTMs connecting pure Transformer attention layers. Our experiments with the 6-layer Transformer show significant BLEU improvements in both WMT 14 English-German / French tasks and the OPUS-100 many-to-many multilingual NMT task, and our deep Transformer experiments demonstrate the effectiveness of depth-wise LSTM on the convergence and performance of deep Transformers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
