arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.02648cs.CLcs.AI
收藏

关于Transformer译码器的子层功能

On the Sub-Layer Functionalities of Transformer Decoder

Oregon State University(俄勒冈州立大学);Tencent AI Lab(腾讯人工智能实验室)

Yilin Yang, Longyue Wang, Shuming Shi, Prasad Tadepalli, Stefan Lee, Zhaopeng Tu

摘要:

已经有大量的努力来解释神经机器翻译(NMT)的基于transformer的编码器-解码器架构的编码器;同时,尽管解码器具有关键作用,但它在很大程度上仍未被研究。在翻译过程中,解码器必须通过考虑来自编码器的源语言文本和在先前步骤中产生的目标语言前缀来预测输出标记。在这项工作中,我们研究了基于transformer的解码器如何利用来自源语言和目标语言的信息-开发一个通用的探测任务,以评估信息如何通过每个解码器层的每个模块传播。我们在三个主要的翻译数据集(WMT En-De,En-Fr和En-Zh)上进行了广泛的实验。我们的分析提供了关于解码器何时何地利用不同来源的见解。基于这些见解,我们证明了每个Transformer解码器层中的残差前馈模块可以以最小的性能损失被丢弃-计算和参数数量的显着减少,从而显着提高训练和推理速度。

英文摘要:

There have been significant efforts to interpret the encoder of Transformer-based encoder-decoder architectures for neural machine translation (NMT); meanwhile, the decoder remains largely unexamined despite its critical role. During translation, the decoder must predict output tokens by considering both the source-language text from the encoder and the target-language prefix produced in previous steps. In this work, we study how Transformer-based decoders leverage information from the source and target languages -- developing a universal probe task to assess how information is propagated through each module of each decoder layer. We perform extensive experiments on three major translation datasets (WMT En-De, En-Fr, and En-Zh). Our analysis provides insight on when and where decoders leverage different sources. Based on these insights, we demonstrate that the residual feed-forward module in each Transformer decoder layer can be dropped with minimal loss of performance -- a significant reduction in computation and number of parameters, and consequently a significant boost to both training and inference speed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑