arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1902.05766cs.CL
收藏

上下文感知自注意网络

Context-Aware Self-Attention Networks

University of Macau(澳门大学);The Chinese University of Hong Kong(香港中文大学);Tencent AI Lab(腾讯人工智能实验室)

Baosong Yang, Jian Li, Derek Wong, Lidia S. Chao, Xing Wang, Zhaopeng Tu

摘要:

自注意模型在并行计算方面表现出了灵活性,在建模长期和短期依赖关系方面也表现出了有效性。然而,它计算表示之间的依赖关系,而不考虑上下文信息,这已被证明是有用的建模在各种自然语言任务中的神经表示之间的依赖关系。在这项工作中,我们专注于通过捕捉丰富的上下文来改善自我注意网络。为了保持自注意力网络的简单性和灵活性,我们建议将查询和关键层的转换上下文化,这些转换用于计算元素之间的相关性。具体来说,我们利用嵌入全局和深层上下文的内部表示,从而避免依赖外部资源。在WMT 14英-德和WMT 17汉英翻译任务上的实验结果证明了所提方法的有效性和通用性。此外,我们进行了广泛的分析,量化的上下文向量如何参与自我注意模型。

英文摘要:

Self-attention model have shown its flexibility in parallel computation and the effectiveness on modeling both long- and short-term dependencies. However, it calculates the dependencies between representations without considering the contextual information, which have proven useful for modeling dependencies among neural representations in various natural language tasks. In this work, we focus on improving self-attention networks through capturing the richness of context. To maintain the simplicity and flexibility of the self-attention networks, we propose to contextualize the transformations of the query and key layers, which are used to calculates the relevance between elements. Specifically, we leverage the internal representations that embed both global and deep contexts, thus avoid relying on external resources. Experimental results on WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed methods. Furthermore, we conducted extensive analyses to quantity how the context vectors participate in the self-attention model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑