带有层归一化的循环Transformer可证明地学习幂方法
Looped Transformers with Layer Normalization Provably Learn the Power Method
AI总结：
本文通过主成分预测任务，证明带有层归一化的循环线性Transformer在梯度下降训练下会收敛到实现幂方法的解，揭示了层归一化带来的算法隐式偏差。
摘要：
Transformer在广泛的应用中取得了显著成功，越来越多的研究表明其部分优势来自于学习和执行算法程序的能力。然而，我们对Transformer如何学习此类算法的理解仍然有限，尤其是在存在层归一化（LN）的情况下。在这项工作中，我们研究主成分预测作为理解带有LN的Transformer训练动态的具体测试平台。我们证明，通过梯度下降训练的带有LN的循环线性Transformer收敛到实现幂方法的解，其中每个自注意力层执行一次幂迭代。值得注意的是，模型仅针对主成分预测进行训练，而非明确监督其实现幂方法。因此，我们的发现揭示了带有LN的循环Transformer的“算法隐式偏差”：主成分预测原则上可以通过多种机制实现，但梯度下降选择了实现幂方法的一种。我们进一步提供了带有和不带有LN的Transformer之间的具体比较：即使有幂迭代的逐层指导，没有LN的Transformer也无法精确学习幂方法，而带有LN的对应Transformer可以，导致主成分预测中可证明的性能差距。据我们所知，我们的结果首次对带有LN的循环和单层Transformer的训练动态进行了理论分析，并阐明了LN在Transformer模型中的作用。
英文摘要：
Transformers have achieved remarkable success across a wide range of applications, and a growing body of work suggests that part of their strength comes from their ability to learn and execute algorithmic procedures. However, our understanding of how transformers learn such algorithms remains limited, especially in the presence of layer normalization (LN). In this work, we study principal component prediction as a concrete testbed for understanding the training dynamics of transformers with LN. We prove that a looped linear transformer with LN, trained by gradient descent, converges to a solution that implements the power method, with each self-attention layer performing one power iteration. Notably, the model is trained only for principal component prediction, rather than being explicitly supervised to implement the power method. Our finding thus reveals an "algorithmic implicit bias" of looped transformers with LN: principal-component prediction can in principle be achieved by many mechanisms, yet gradient descent selects one that realizes the power method. We further provide a concrete comparison between transformers with and without LN: even with layerwise guidance from power iterations, a transformer without LN cannot exactly learn the power method, whereas the corresponding transformer with LN can, leading to a provable performance gap in principal component prediction. Our results provide, to our knowledge, the first theoretical analysis of the training dynamics of looped and single-layer transformers with LN, and shed light on the role of LN in transformer models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
