arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.08993math.DGcs.AI
收藏

大型语言模型的 token 空间结构

The structure of the token space for large language models

American University(美利坚大学);Galois, Inc.(Galois公司)

Michael Robinson, Sourya Dey, Shauna Sweet

AI总结:

本文通过估计 token 子空间的维度和 Ricci 标量曲率,发现其结构为分层流形且曲率显著为负,并指出该几何特征与模型生成流畅性相关,为理解大模型行为与局限提供了第一性原理视角。

摘要:

大型语言模型通过将话语片段(tokens)拟合到高维环境潜在空间中来编码自然语言中存在的相关结构,模型随后在该空间上进行操作。我们断言,为了发展对大型语言模型行为和局限性的基础性第一性原理理解,理解该 token 子空间的拓扑和几何结构至关重要。在本文中,我们提出了 token 子空间维度和 Ricci 标量曲率的估计量,并将其应用于三个中等规模的开源大型语言模型:GPT2、LLEMMA7B 和 MISTRAL7B。在所有三个模型中,使用这些测量,我们发现 token 子空间不是流形,而是分层流形,在各个单独的层上,Ricci 曲率显著为负。我们额外发现维度和曲率与模型的生成流畅性相关,这表明这些发现对模型行为具有启示意义。

英文摘要:

Large language models encode the correlational structure present in natural language by fitting segments of utterances (tokens) into a high dimensional ambient latent space upon which the models then operate. We assert that in order to develop a foundational, first-principles understanding of the behavior and limitations of large language models, it is crucial to understand the topological and geometric structure of this token subspace. In this article, we present estimators for the dimension and Ricci scalar curvature of the token subspace, and apply it to three open source large language models of moderate size: GPT2, LLEMMA7B, and MISTRAL7B. In all three models, using these measurements, we find that the token subspace is not a manifold, but is instead a stratified manifold, where on each of the individual strata, the Ricci curvature is significantly negative. We additionally find that the dimension and curvature correlate with generative fluency of the models, which suggest that these findings have implications for model behavior.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑