注意力层可证明解决单位置回归问题
Attention layers provably solve single-location regression
AI总结：
针对注意力机制缺乏理论理解的问题，提出单位置回归任务及简化自注意力预测器，证明其渐近最优性并分析训练动态，凸显处理稀疏信息与线性结构的能力。
摘要：
基于注意力的模型，如Transformer，在各种任务中表现出色，但缺乏全面的理论理解，尤其是在逐词稀疏性和内部线性表示方面。为弥补这一空白，我们引入了单位置回归任务，其中序列中仅有一个标记决定输出，其位置是一个潜在随机变量，可通过输入的线性投影检索。为解决此任务，我们提出了一种专用预测器，该预测器实际上是非线性自注意力层的简化版本。我们通过展示其渐近贝叶斯最优性并分析其训练动态来研究其理论性质。特别是，尽管问题具有非凸性，该预测器仍能有效学习底层结构。这项工作凸显了注意力机制处理稀疏标记信息和内部线性结构的能力。
英文摘要：
Attention-based models, such as Transformer, excel across various tasks but lack a comprehensive theoretical understanding, especially regarding token-wise sparsity and internal linear representations. To address this gap, we introduce the single-location regression task, where only one token in a sequence determines the output, and its position is a latent random variable, retrievable via a linear projection of the input. To solve this task, we propose a dedicated predictor, which turns out to be a simplified version of a non-linear self-attention layer. We study its theoretical properties, by showing its asymptotic Bayes optimality and analyzing its training dynamics. In particular, despite the non-convex nature of the problem, the predictor effectively learns the underlying structure. This work highlights the capacity of attention mechanisms to handle sparse token information and internal linear structures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
