arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2005.09561cs.LGstat.ML
收藏

没有概率笼的归一化注意

Normalized Attention Without Probability Cage

ETH Zurich(苏黎世联邦理工学院)

Oliver Richter, Roger Wattenhofer

摘要:

注意力架构被广泛使用;它们最近随着Transformers重新流行起来,产生了一系列最先进的结果。然而,softmax-attention的几何含义在很大程度上仍未被探索。在这项工作中,我们强调了限制注意力权重的概率单纯形和由此产生的值向量的凸包的局限性。我们发现,Transformers是序列长度依赖的偏向令牌隔离在初始化和对比Transformers简单的最大和总和池-两个强基线很少报道。我们建议用归一化来取代自注意力中的softmax,从而产生一个超参数和数据偏差鲁棒的、普遍适用的架构。我们通过来自25,000多个训练模型的经验结果来支持我们的见解。所有结果和实现都是可用的。

英文摘要:

Attention architectures are widely used; they recently gained renewed popularity with Transformers yielding a streak of state of the art results. Yet, the geometrical implications of softmax-attention remain largely unexplored. In this work we highlight the limitations of constraining attention weights to the probability simplex and the resulting convex hull of value vectors. We show that Transformers are sequence length dependent biased towards token isolation at initialization and contrast Transformers to simple max- and sum-pooling - two strong baselines rarely reported. We propose to replace the softmax in self-attention with normalization, yielding a hyperparameter and data-bias robust, generally applicable architecture. We support our insights with empirical results from more than 25,000 trained models. All results and implementations are made available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑