arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.09414cs.LG
收藏

聚类与对齐:理解模加法中的训练动态

Clustering and Alignment: Understanding the Training Dynamics in Modular Addition

ETH Zürich(苏黎世联邦理工学院)

Tiberiu Musat

AI总结:

本文研究小型神经网络在模加法任务中的训练动态,用嵌入对的聚类与对齐作用力解释网格和圆形结构的涌现,并通过等效粒子模拟及权重衰减机制分析加以验证。

摘要:

近期研究表明,神经网络会为许多简单问题学习可解释的算法。然而,人们对这些算法在训练过程中如何涌现仍知之甚少。本文中,我研究了一个具有二维嵌入的小型神经网络在模加法问题上的训练动态。我观察到,嵌入向量倾向于组织成两类结构:网格和圆。我研究了这些结构,并将它们的涌现解释为嵌入对所表现出的两种简单趋势的结果:聚类与对齐。我将这些趋势以显式公式表示为不同嵌入对之间的相互作用力。为证明我的公式能够完全解释这些结构的涌现,我构建了一个等效的粒子模拟,并在其中展示了相同结构会涌现出来。我讨论了权重衰减在该设置中的作用,并揭示了一种连接正则化与训练动态的新机制。为支持我的发现,我还发布了一个可在 https://modular-addition.vercel.app/ 获取的交互式演示。

英文摘要:

Recent studies have revealed that neural networks learn interpretable algorithms for many simple problems. However, little is known about how these algorithms emerge during training. In this article, I study the training dynamics of a small neural network with 2-dimensional embeddings on the problem of modular addition. I observe that embedding vectors tend to organize into two types of structures: grids and circles. I study these structures and explain their emergence as a result of two simple tendencies exhibited by pairs of embeddings: clustering and alignment. I propose explicit formulae for these tendencies as interaction forces between different pairs of embeddings. To show that my formulae can fully account for the emergence of these structures, I construct an equivalent particle simulation where I show that identical structures emerge. I discuss the role of weight decay in my setup and reveal a new mechanism that links regularization and training dynamics. To support my findings, I also release an interactive demo available at https://modular-addition.vercel.app/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑