arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2508.00331cs.LG
收藏

语言模型的胚胎学

Embryology of a Language Model

Timaeus(蒂迈欧)

George Wang, Garrett Baker, Andrew Gordon, Daniel Murfet

AI总结:

本研究提出胚胎学方法,将UMAP用于磁化率矩阵可视化语言模型训练中的结构发展,既追踪已知特征形成,还发现新结构,为研究复杂神经网络发育原理提供了强大的整体性分析视角。

摘要:

理解语言模型如何形成其内部计算结构是深度学习科学中的核心问题。虽然源自统计物理学的磁化率(susceptibilities)提供了一种很有前景的分析工具,但它们在可视化网络组织结构方面的全部潜力仍未被发掘。在本研究中,我们提出了一种胚胎学方法,将UMAP应用于磁化率矩阵,以可视化模型在训练过程中的结构发展。我们的可视化结果揭示了一个清晰的“身体规划”的出现,既追踪了诱导回路等已知特征的形成过程,还发现了此前未知的结构,比如专门用于计数空格标记的“间距鳍”。本研究表明,磁化率分析可以超越验证的范畴,用于发现新的机制,为研究复杂神经网络的发育原理提供了一种强大的整体性视角。

英文摘要:

Understanding how language models develop their internal computational structure is a central problem in the science of deep learning. While susceptibilities, drawn from statistical physics, offer a promising analytical tool, their full potential for visualizing network organization remains untapped. In this work, we introduce an embryological approach, applying UMAP to the susceptibility matrix to visualize the model's structural development over training. Our visualizations reveal the emergence of a clear ``body plan,'' charting the formation of known features like the induction circuit and discovering previously unknown structures, such as a ``spacing fin'' dedicated to counting space tokens. This work demonstrates that susceptibility analysis can move beyond validation to uncover novel mechanisms, providing a powerful, holistic lens for studying the developmental principles of complex neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑