探索层归一化对零样本神经机器翻译的影响
Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation
AI总结：
本文研究了层归一化对零样本翻译的影响，发现残差连接后归一化（PostNorm）在54个翻译方向上显著优于默认的PreNorm，最高提升12.3 BLEU，强调了ZST中需谨慎选择LayerNorm设置。
摘要：
本文研究了层归一化（LayerNorm）对零样本翻译（ZST）的影响。近期针对ZST的研究通常采用Transformer架构作为骨干网络，并将层输入处的LayerNorm（PreNorm）作为默认设置。然而，Xu等人（2019）揭示PreNorm存在过拟合训练数据的风险。基于此，我们假设PreNorm可能过拟合监督方向，从而对ZST的泛化能力较低。通过在OPUS、IWSLT和Europarl数据集上对54个ZST方向进行实验，我们证明残差连接后进行LayerNorm的原始Transformer设置（PostNorm）始终优于PreNorm，最高可达12.3个BLEU点。随后，我们通过分析PreNorm和PostNorm之间非目标率和结构差异，研究了性能差异。本研究强调了对ZST的LayerNorm设置需要谨慎考虑。
英文摘要：
This paper studies the impact of layer normalization (LayerNorm) on zero-shot translation (ZST). Recent efforts for ZST often utilize the Transformer architecture as the backbone, with LayerNorm at the input of layers (PreNorm) set as the default. However, Xu et al. (2019) has revealed that PreNorm carries the risk of overfitting the training data. Based on this, we hypothesize that PreNorm may overfit supervised directions and thus have low generalizability for ZST. Through experiments on OPUS, IWSLT, and Europarl datasets for 54 ZST directions, we demonstrate that the original Transformer setting of LayerNorm after residual connections (PostNorm) consistently outperforms PreNorm by up to 12.3 BLEU points. We then study the performance disparities by analyzing the differences in off-target rates and structural variations between PreNorm and PostNorm. This study highlights the need for careful consideration of the LayerNorm setting for ZST.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
