深度神经机器翻译模型的共生网络联合训练
Joint-training on Symbiosis Networks for Deep Nueral Machine Translation models
摘要：
深度编码器已被证明在改善神经机器翻译（NMT）系统方面是有效的，但当编码器层数超过18时，它达到了翻译质量的上限。更糟糕的是，更深层次的网络会消耗大量内存，从而无法有效地进行训练。在本文中，我们提出了共生网络，它包括一个完整的网络作为共生主网（M-Net）和另一个共享的子网络与共生子网（S-Net）的结构相同，但较少的层。我们采用Symbiosis Networks on Transformer-deep（m-n）架构，在NMT中定义了M-Net和S-Net之间的正则化损失$\mathcal{L}_τ$。我们将联合训练应用于共生网络，旨在提高M-Net的性能。我们提出的训练策略在WMT'14 EN->DE，DE->EN和EN->FR任务的经典训练下，将Transformer-deep（12-6）提高了0.61，0.49和0.69 BLEU。此外，我们的Transformer-deep（12-6）甚至优于经典的Transformer-deep（18-6）。
英文摘要：
Deep encoders have been proven to be effective in improving neural machine translation (NMT) systems, but it reaches the upper bound of translation quality when the number of encoder layers exceeds 18. Worse still, deeper networks consume a lot of memory, making it impossible to train efficiently. In this paper, we present Symbiosis Networks, which include a full network as the Symbiosis Main Network (M-Net) and another shared sub-network with the same structure but less layers as the Symbiotic Sub Network (S-Net). We adopt Symbiosis Networks on Transformer-deep (m-n) architecture and define a particular regularization loss $\mathcal{L}_τ$ between the M-Net and S-Net in NMT. We apply joint-training on the Symbiosis Networks and aim to improve the M-Net performance. Our proposed training strategy improves Transformer-deep (12-6) by 0.61, 0.49 and 0.69 BLEU over the baselines under classic training on WMT'14 EN->DE, DE->EN and EN->FR tasks. Furthermore, our Transformer-deep (12-6) even outperforms classic Transformer-deep (18-6).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
