arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.01545cs.LGphysics.data-an
收藏

大型语言模型中的思维线

Lines of Thought in Large Language Models

School of Civil and Environmental Engineering(土木与环境工程学院);Cornell University(康奈尔大学);Department of Physics(物理系);Department of Mathematics(数学系);Imperial College London(伦敦帝国理工学院);Center for Applied Mathematics(应用数学中心)

Raphaël Sarfati, Toni J. B. Liu, Nicolas Boullé, Christopher J. Earls

AI总结:

本文研究大型语言模型中 transformer 层形成的高维“思维线”轨迹,发现其聚集于低维非欧流形,并可用少量参数的随机方程近似,从而揭示模型复杂性的简化统计结构。

摘要:

大型语言模型通过在连续 transformer 层的作用下,将向量化的文本片段(提示词)沿伴随的嵌入空间传输,来实现下一 token 预测。由此产生的高维轨迹实现了不同的上下文化或“思考”步骤,并完全决定输出概率分布。我们旨在刻画这些“思维线”集合的统计特性。我们观察到,独立轨迹沿一个低维、非欧几里得流形聚集,而且其路径可以用一个从数据中提取的、仅有少量参数的随机方程很好地近似。我们认为值得注意的是,如此大型模型的巨大复杂性可以被简化为一种简单得多的形式,并且我们也思考了其相关意义。

英文摘要:

Large Language Models achieve next-token prediction by transporting a vectorized piece of text (prompt) across an accompanying embedding space under the action of successive transformer layers. The resulting high-dimensional trajectories realize different contextualization, or 'thinking', steps, and fully determine the output probability distribution. We aim to characterize the statistical properties of ensembles of these 'lines of thought.' We observe that independent trajectories cluster along a low-dimensional, non-Euclidean manifold, and that their path can be well approximated by a stochastic equation with few parameters extracted from data. We find it remarkable that the vast complexity of such large models can be reduced to a much simpler form, and we reflect on implications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑