arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.08235cs.LGcs.CL
收藏

PairConnect:一种计算效率高的MLP注意力替代方案

PairConnect: A Compute-Efficient MLP Alternative to Attention

Rice University(莱斯大学)

Zhaozhuo Xu, Minghao Yan, Junyan Zhang, Anshumali Shrivastava

摘要:

Transformer模型在自然语言处理方面表现出了优越的性能。Transformer中的点积自我注意力使我们能够对单词之间的交互进行建模。然而,这种建模伴随着显著的计算开销。在这项工作中,我们重新审视了与Transformer相关的内存-计算权衡,特别是多头注意力,并显示了一个内存繁重但计算效率更高的替代方案Transformer。我们的建议,表示为PairConnect,多层感知器(MLP),模型的两两之间的相互作用,明确的两两词嵌入。因此,PairConnect用一个简单的嵌入查找代替了自点积。我们在数学上表明,尽管是一个MLP,我们的计算效率PairConnect严格比Transformer更有表现力。我们在语言建模任务上的实验表明,PairConnect可以实现与Transformer相当的结果,同时显著降低与推理相关的计算成本。

英文摘要:

Transformer models have demonstrated superior performance in natural language processing. The dot product self-attention in Transformer allows us to model interactions between words. However, this modeling comes with significant computational overhead. In this work, we revisit the memory-compute trade-off associated with Transformer, particularly multi-head attention, and show a memory-heavy but significantly more compute-efficient alternative to Transformer. Our proposal, denoted as PairConnect, a multilayer perceptron (MLP), models the pairwise interaction between words by explicit pairwise word embeddings. As a result, PairConnect substitutes self dot product with a simple embedding lookup. We show mathematically that despite being an MLP, our compute-efficient PairConnect is strictly more expressive than Transformer. Our experiment on language modeling tasks suggests that PairConnect could achieve comparable results with Transformer while reducing the computational cost associated with inference significantly.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑