Transformers中层权矩阵的贪婪排序改进了翻译
Greedy Ordering of Layer Weight Matrices in Transformers Improves Translation
摘要:
先前的工作试图理解基于变换器的编码器-解码器架构在多头注意力和前馈子层的级别上的内部结构和功能。解释集中在编码器和解码器上,以及自注意、交叉注意和前馈子层的组合可能性。然而,如果不检查低级结构,人们对子层重新排序背后的动机的了解就有限。我们是否可以深入到子层抽象和置换层权重矩阵,以提高翻译质量?我们提出了AEIUOrder,通过重尾自正则化(HT-SR)度量测量编码器中的良好训练来对层权重矩阵进行重排序,并相应地对解码器矩阵进行排序。我们的研究结果表明,greatness重新排序层权重矩阵,以最大限度地提高总的良好训练,有利于模型学习表示和更有效地生成翻译。
英文摘要:
Prior work has attempted to understand the internal structures and functionalities of Transformer-based encoder-decoder architectures on the level of multi-head attention and feed-forward sublayers. Interpretations have focused on the encoder and decoder, along with the combinatorial possibilities of the self-attention, cross-attention, and feed-forward sublayers. However, without examining the low-level structures, one gains limited understanding of the motivation behind sublayer reordering. Could we dive into the sublayer abstraction and permute layer weight matrices to improve the quality of translation? We propose AEIUOrder to greedily reorder layer weight matrices in the encoder by their well-trainedness, as measured by Heavy-Tailed Self-Regularization (HT-SR) metrics, and order the decoder matrices correspondingly. Our results suggest that greedily reordering layer weight matrices to maximize Total well-trainedness facilitates the model to learn representations and generate translations more effectively.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。