双路径网络在序列到序列学习中的应用
Double Path Networks for Sequence to Sequence Learning
摘要:
基于编码器-解码器的序列到序列学习(S2 S)近年来取得了显著的进展。在编码器/解码器中使用了不同的网络架构。其中,卷积神经网络(CNN)和自注意网络(SAN)是比较突出的两种。这两种架构实现了类似的性能,但使用非常不同的方式来编码和解码上下文:CNN使用卷积层来关注序列的局部连接,而SAN使用自注意层来关注全局语义。在这项工作中,我们提出了双路径网络序列到序列学习(DPN-S2 S),它利用两个模型的优点,通过使用双路径信息融合。在编码步骤中,我们开发了一个双路径架构,分别用卷积层和自注意层来维护来自不同路径的信息。为了有效地使用编码的上下文,我们开发了一个交叉注意模块与门控,并使用它来自动拾取解码步骤中所需的信息。通过深度整合这两条路径,交叉注意,这两种类型的信息结合起来,很好地利用。实验表明,我们提出的方法可以显着提高性能的序列到序列的学习状态的最先进的系统。
英文摘要:
Encoder-decoder based Sequence to Sequence learning (S2S) has made remarkable progress in recent years. Different network architectures have been used in the encoder/decoder. Among them, Convolutional Neural Networks (CNN) and Self Attention Networks (SAN) are the prominent ones. The two architectures achieve similar performances but use very different ways to encode and decode context: CNN use convolutional layers to focus on the local connectivity of the sequence, while SAN uses self-attention layers to focus on global semantics. In this work we propose Double Path Networks for Sequence to Sequence learning (DPN-S2S), which leverage the advantages of both models by using double path information fusion. During the encoding step, we develop a double path architecture to maintain the information coming from different paths with convolutional layers and self-attention layers separately. To effectively use the encoded context, we develop a cross attention module with gating and use it to automatically pick up the information needed during the decoding step. By deeply integrating the two paths with cross attention, both types of information are combined and well exploited. Experiments show that our proposed method can significantly improve the performance of sequence to sequence learning over state-of-the-art systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。