论注意的规律性
On the Regularity of Attention
摘要：
注意力是现代神经网络的一个强大组成部分，它跨越了许多领域。在本文中，我们试图量化的规律性（即平滑量）的注意力操作。为了实现这一目标，我们提出了一个新的数学框架，使用测量理论和积分算子模型的注意。我们表明，这个框架是符合通常的定义，它抓住了注意力的基本属性。然后，我们使用这个框架来证明，在紧域上，注意操作是Lipschitz连续的，并提供了一个估计的Lipschitz常数。此外，通过专注于一个特定类型的注意力，我们扩展这些Lipschitz连续性的结果，非紧域。我们还讨论了规则性对NLP模型的影响，以及对可逆和无限深度网络的应用。
英文摘要：
Attention is a powerful component of modern neural networks across a wide variety of domains. In this paper, we seek to quantify the regularity (i.e. the amount of smoothness) of the attention operation. To accomplish this goal, we propose a new mathematical framework that uses measure theory and integral operators to model attention. We show that this framework is consistent with the usual definition, and that it captures the essential properties of attention. Then we use this framework to prove that, on compact domains, the attention operation is Lipschitz continuous and provide an estimate of its Lipschitz constant. Additionally, by focusing on a specific type of attention, we extend these Lipschitz continuity results to non-compact domains. We also discuss the effects regularity can have on NLP models, and applications to invertible and infinitely-deep networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
