arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.23239cs.LGstat.ML
收藏

向理解变换器在学习随机游走中的能力

Towards Understanding Transformers in Learning Random Walks

School of Computing & Data Science, The University of Hong Kong(计算与数据科学学院,香港大学)

Wei Shi, Yuan Cao

AI总结:

本文研究了变换器在学习随机游走中的能力和可解释性,证明单层变换器在预测随机游走时能达到最优准确性,并揭示了其可解释性机制。

摘要:

变换器在各种应用中已证明具有高度有效性,尤其是在处理序列数据如自然语言和时间序列时。然而,变换器模型往往缺乏清晰的可解释性,变换器的成功在理论上并未得到充分理解。在本文中,我们研究了变换器在学习一组经典统计模型(即圆上的随机游走)中的能力和可解释性。我们理论证明,经过梯度下降训练后,单层变换器模型可以在预测随机游走时达到最优准确性。重要的是,我们的分析揭示了训练后的模型是可解释的:训练后的softmax注意力充当一个token选择器,专注于直接父状态;随后,值矩阵执行一步概率转移,根据此父状态预测下一个状态的位置。我们还展示了某些未被我们的理论覆盖的边缘情况确实为失败案例,证明我们的理论条件是紧致的。通过研究这些成功和失败案例,揭示了在某些简单的任务中,即使超出随机游走,小初始化的梯度下降可能失败或难以收敛到一个好的解。实验被进行以支持我们的理论发现。

英文摘要:

Transformers have proven highly effective across various applications, especially in handling sequential data such as natural languages and time series. However, transformer models often lack clear interpretability, and the success of transformers has not been well understood in theory. In this paper, we study the capability and interpretability of transformers in learning a family of classic statistical models, namely random walks on circles. We theoretically demonstrate that, after training with gradient descent, a one-layer transformer model can achieve optimal accuracy in predicting random walks. Importantly, our analysis reveals that the trained model is interpretable: the trained softmax attention serves as a token selector, focusing on the direct parent state; subsequently, the value matrix executes a one-step probability transition to predict the location of the next state based on this parent state. We also show that certain edge cases not covered by our theory are indeed failure cases, demonstrating that our theoretical conditions are tight. By investigating these success and failure cases, it is revealed that gradient descent with small initialization may fail or struggle to converge to a good solution in certain simple tasks even beyond random walks. Experiments are conducted to support our theoretical findings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑