Poly-attention：一种用于高阶自注意力的通用方案
Poly-attention: a general scheme for higher-order self-attention
AI总结：
本文提出了一种新的多注意力机制，能够在二次时间内精确计算并支持任意数量函数的组合，解决了传统自注意力机制在处理多阶任务时的效率问题。
摘要：
Transformer模型中的自注意力机制能够有效建模令牌之间的成对交互。然而，许多近期研究表明，它无法执行涉及三个相关令牌的检测任务，或需要多个输入令牌参考以生成结果的组合性任务。一些更高维度的自注意力替代方案已被提出，包括高阶注意力和Strassen注意力，这些方案可以执行一些多阶任务，但以更慢的超二次运行时间为代价。在本工作中，我们定义了自注意力的广泛泛化类，称为多注意力机制。我们的机制可以整合任意高阶（张量）计算以及任意输入令牌之间的关系结构，并且包括上述替代方案作为特殊情况。然后，我们系统地研究了它们的计算复杂性和表示强度，包括给出新的算法和匹配的时间复杂度下界，以及精确计算注意力矩阵以及近似计算的时间复杂度下界。我们还精确确定了它们各自能执行的多阶任务。我们的结果给出了这些机制不同需求之间的有趣权衡，包括表达性机制与模型中系数大小之间的紧密关系，使得机制可以以近线性时间近似。值得注意的是，我们提供了一种可以在二次时间内精确计算的注意力机制，并且可以为任何固定数量的函数执行函数组合。先前的机制，即使仅用于组合两个函数，也只能在超二次时间内计算，我们的新下界表明，对于它们的更快算法是不可能的。
英文摘要：
The self-attention mechanism, at the heart of the Transformer model, is able to effectively model pairwise interactions between tokens. However, numerous recent works have shown that it is unable to perform basic tasks involving detecting triples of correlated tokens, or compositional tasks where multiple input tokens need to be referenced to generate a result. Some higher-dimensional alternatives to self-attention have been proposed to address this, including higher-order attention and Strassen attention, which can perform some of these polyadic tasks in exchange for slower, superquadratic running times. In this work, we define a vast class of generalizations of self-attention, which we call poly-attention mechanisms. Our mechanisms can incorporate arbitrary higher-order (tensor) computations as well as arbitrary relationship structures between the input tokens, and they include the aforementioned alternatives as special cases. We then systematically study their computational complexity and representational strength, including giving new algorithms and matching complexity-theoretic lower bounds on the time complexity of computing the attention matrix exactly as well as approximately, and tightly determining which polyadic tasks they can each perform. Our results give interesting trade-offs between different desiderata for these mechanisms, including a tight relationship between how expressive a mechanism is, and how large the coefficients in the model may be so that the mechanism can be approximated in almost-linear time. Notably, we give a new attention mechanism which can be computed exactly in quadratic time, and which can perform function composition for any fixed number of functions. Prior mechanisms, even for just composing two functions, could only be computed in superquadratic time, and our new lower bounds show that faster algorithms for them are not possible.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
