arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.14649cs.CL
收藏

用于神经机器翻译的Capsule-Transformer

Capsule-Transformer for Neural Machine Translation

Shanghai Jiao Tong University(上海交通大学)

Sufeng Duan, Juncheng Cao, Hai Zhao

摘要:

Transformer从其多头自注意网络(SAN)的关键设计中受益匪浅,该网络通过将给定的输入转换到不同的子空间来从各种角度提取信息。然而,其简单的线性转换聚合策略仍然可能无法完全捕获更深层次的上下文信息。因此,在本文中,我们提出了胶囊变压器,它扩展了线性变换到一个更一般的胶囊路由算法,通过将SAN作为一个特殊的情况下胶囊网络。因此,所产生的胶囊变压器是能够获得一个更好的注意力分布表示的输入序列,通过不同的头部和单词之间的信息聚合。具体来说,我们看到SAN中的注意力权重组是低层胶囊。通过应用迭代胶囊路由算法,它们可以被进一步聚合成包含更深的上下文信息的高层胶囊。在广泛使用的机器翻译数据集上的实验结果表明,我们提出的胶囊-Transformer显著优于强Transformer基线。

英文摘要:

Transformer hugely benefits from its key design of the multi-head self-attention network (SAN), which extracts information from various perspectives through transforming the given input into different subspaces. However, its simple linear transformation aggregation strategy may still potentially fail to fully capture deeper contextualized information. In this paper, we thus propose the capsule-Transformer, which extends the linear transformation into a more general capsule routing algorithm by taking SAN as a special case of capsule network. So that the resulted capsule-Transformer is capable of obtaining a better attention distribution representation of the input sequence via information aggregation among different heads and words. Specifically, we see groups of attention weights in SAN as low layer capsules. By applying the iterative capsule routing algorithm they can be further aggregated into high layer capsules which contain deeper contextualized information. Experimental results on the widely-used machine translation datasets show our proposed capsule-Transformer outperforms strong Transformer baseline significantly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑