Transformer 可证明地学习稀疏 token 选择，而全连接网络不能
Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot
AI总结：
本文研究稀疏 token 选择任务，证明一层 Transformer 可通过梯度下降学习并实现长度泛化，同时将 FCN 的下界强化至平均情况，建立二者的算法分离。
摘要：
Transformer 架构因其在选择和组合结构信息方面的卓越能力，已在多种深度学习场景中占据主导。受这些能力启发，Sanford 等人提出了稀疏 token 选择任务：在该任务上，Transformer 表现出色，而全连接网络（FCN）在最坏情况下会失败。在此基础上，我们将 FCN 的下界强化到平均情况设定，并建立了 Transformer 相对于 FCN 的算法分离。具体而言，经梯度下降训练的一层 Transformer 可证明地学会稀疏 token 选择任务，并且令人惊讶地展现出强大的分布外长度泛化能力。我们提供了实证仿真来证明理论发现的合理性。
英文摘要：
The transformer architecture has prevailed in various deep learning settings due to its exceptional capabilities to select and compose structural information. Motivated by these capabilities, Sanford et al. proposed the sparse token selection task, in which transformers excel while fully-connected networks (FCNs) fail in the worst case. Building upon that, we strengthen the FCN lower bound to an average-case setting and establish an algorithmic separation of transformers over FCNs. Specifically, a one-layer transformer trained with gradient descent provably learns the sparse token selection task and, surprisingly, exhibits strong out-of-distribution length generalization. We provide empirical simulations to justify our theoretical findings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
