大型语言模型隐藏状态的超双曲性图谱

中文摘要 AI 辅助

大型语言模型（LLM）的隐藏状态是普通向量，但这些向量之间的距离可能仍表现出层级结构。据我们所知，本文首次系统研究了当代LLM中提示词令牌的隐藏状态是否表现出Gromov超双曲性（GH），这是一种基于距离的树状性度量。我们使用来自十个开放权重模型在MATH500、HumanEval、WinoGrande和TruthfulQA上的818,904个样本-层测量，构建了一个涵盖四个轴的GH图谱：参数规模、层深度、模型家族和输入领域。最清晰的模式是深度而非规模：中间层通常形成高相对超双曲性的平台，而最终层往往变得显著更接近树状。规模效应较弱且非单调，匹配的7/8B模型家族差异显著，领域与模型专业化相互作用。这些发现使GH作为实用诊断工具具有价值：它展示了层级距离结构出现的位置、专业化如何改变它，以及哪些模型-层-领域比较值得更深入分析。

英文摘要

LLM hidden states are ordinary vectors, but the distances among those vectors may still show hierarchical structure. To our knowledge, this paper is the first systematic study of whether prompt-token hidden states in contemporary LLMs exhibit Gromov Hyperbolicity (GH), a distance-based measure of tree-likeness. Using 818,904 sample-layer measurements from ten open-weight models across MATH500, HumanEval, WinoGrande, and TruthfulQA, we build a GH map over four axes: parameter scale, layer depth, model family, and input domain. The clearest pattern is depth, not scale: middle layers usually form a high-relative-hyperbolicity plateau, while final layers often become substantially more tree-like. Scale effects are weak and non-monotonic, matched 7/8B model families differ strongly, and domains interact with model specialization. These findings make GH useful as a practical diagnostic: it shows where hierarchical distance structure appears, how specialization changes it, and which model-layer-domain comparisons deserve closer analysis.

发表机构

• Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（未来区块链与隐私计算北京高精尖创新中心）
• School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）
• South China University of Technology（华南理工大学）

机构由 AI 辅助整理，请以论文原文为准。