位置注意力：算法计算的表达能力与可学习性
Positional Attention: Expressivity and Learnability of Algorithmic Computation
AI总结：
本文提出仅依赖位置编码的位置注意力 Transformer，证明其具备并行计算模型的表达力但有对数深度代价，并分析其可学习性、样本复杂度权衡与分布外表现。
摘要：
人们对神经网络执行算法任务（例如算术、汇总统计和排序）的能力日益关注。本工作的目标是更好地理解 Transformers 中注意力在算法执行中的作用。已有研究借助并行计算模型，从理论和实证两方面考察了注意力对算法执行的重要性。值得注意的是，许多并行算法仅使用位置信息在处理器之间进行通信。受这一观察启发，我们研究 Transformers 如何利用位置注意力执行算法，其中注意力权重完全依赖于位置编码。我们证明，具有位置注意力的 Transformers（位置 Transformers）保持了与并行计算模型相同的表达能力，但相对于输入长度会产生对数深度代价。我们分析了它们的分布内可学习性，并探讨位置注意力中的参数范数如何影响样本复杂度。结果表明，位置 Transformers 引入了一种学习权衡：尽管它们在参数范数方面表现出更好的理论依赖性，但某些任务可能需要更多层，而这反过来又会增加样本复杂度。最后，我们通过实证探究位置 Transformers 的分布外性能，发现它们在底层算法解依赖位置信息的任务中表现良好。
英文摘要：
There is a growing interest in the ability of neural networks to execute algorithmic tasks (e.g., arithmetic, summary statistics, and sorting). The goal of this work is to better understand the role of attention in Transformers for algorithmic execution. Its importance for algorithmic execution has been studied theoretically and empirically using parallel computational models. Notably, many parallel algorithms communicate between processors solely using positional information. Inspired by this observation, we investigate how Transformers can execute algorithms using positional attention, where attention weights depend exclusively on positional encodings. We prove that Transformers with positional attention (positional Transformers) maintain the same expressivity of parallel computational models, incurring a logarithmic depth cost relative to the input length. We analyze their in-distribution learnability and explore how parameter norms in positional attention affect sample complexity. Our results show that positional Transformers introduce a learning trade-off: while they exhibit better theoretical dependence on parameter norms, certain tasks may require more layers, which can, in turn, increase sample complexity. Finally, we empirically explore the out-of-distribution performance of positional Transformers and find that they perform well in tasks where their underlying algorithmic solution relies on positional information.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
