arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.22301cs.LGcs.AI
收藏

大型语言模型中的潜在语义流形

Latent Semantic Manifolds in Large Language Models

Qatar University(卡塔尔大学)

Mohamed A. Mabrok

AI总结:

研究大型语言模型中隐藏状态作为流形点的几何解释,定义表达性差距并验证其在不同架构中的表现。

摘要:

大型语言模型(LLMs)在连续向量空间中进行内部计算,却生成离散令牌——这种根本性不匹配的几何后果尚不明确。我们开发了一个数学框架,将LLM隐藏状态解释为潜在语义流形上的点:一个配备Fisher信息度量的Riemann子流形,其中令牌对应于划分流形的Voronoi区域。我们定义了表达性差距,即词汇离散化导致的语义失真度量,并证明了两个定理:任何有限词汇的失真下界,以及通过共面积公式得出的表达性差距线性体积缩放定律。我们验证了六个transformer架构(124M-1.5B参数)中的这些预测,确认了普遍的hourglass内在维度轮廓、平滑曲率结构以及线性差距缩放(斜率0.87-1.12,R²>0.985)。模型中的边缘分布揭示了不变于尺度的边界近邻表示的持久硬核,提供了困惑度的几何分解。我们讨论了架构设计、模型压缩、解码策略和缩放定律的启示。

英文摘要:

Large Language Models (LLMs) perform internal computations in continuous vector spaces yet produce discrete tokens -- a fundamental mismatch whose geometric consequences remain poorly understood. We develop a mathematical framework that interprets LLM hidden states as points on a latent semantic manifold: a Riemannian submanifold equipped with the Fisher information metric, where tokens correspond to Voronoi regions partitioning the manifold. We define the expressibility gap, a geometric measure of the semantic distortion from vocabulary discretization, and prove two theorems: a rate-distortion lower bound on distortion for any finite vocabulary, and a linear volume scaling law for the expressibility gap via the coarea formula. We validate these predictions across six transformer architectures (124M-1.5B parameters), confirming universal hourglass intrinsic dimension profiles, smooth curvature structure, and linear gap scaling with slopes 0.87-1.12 (R^2 > 0.985). The margin distribution across models reveals a persistent hard core of boundary-proximal representations invariant to scale, providing a geometric decomposition of perplexity. We discuss implications for architecture design, model compression, decoding strategies, and scaling laws

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑