发表机构
Massachusetts Institute of Technology（麻省理工学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究探究LLMs隐态推理中连续混合坍缩的原因，结合理论与实证识别出三个失效来源，验证了相关动力学预测，指出精确保留多成分混合需依赖上下文的校正。
AI 中文摘要
大语言模型（LLMs）的隐态推理方法用连续状态（如词元嵌入的加权混合）替代离散中间词元，以保留多种可能的推理方向而非局限于单一方向。然而预训练语言模型常无法保留这些混合状态，我们结合理论分析与对多种模型的可控实证研究探究其原因，识别出三个独立且不同的失效来源：第一，Transformer架构已会扭曲混合几何结构，训练会大幅放大该效应；第二，即便模型以完美线性方式传递混合状态，softmax输出与自回归反馈构成的动力学系统也会放大微小差异直至混合某一成分占优，或压缩不同混合状态直至无法区分；我们通过实证验证了该理论预测：收缩与放大间的观测转变发生在我们分析得出的理论阈值附近，且预训练模型的推演结果主要处于放大侧；最后，我们将结论推广至多成分混合，表明精确保留通常需要依赖上下文的校正，所需维度会随成分数量增长。
英文摘要
LLMs latent-state reasoning methods replace discrete intermediate tokens with continuous states, such as weighted mixtures of token embeddings, to retain multiple possible reasoning directions rather than committing to one. Yet pretrained language models often fail to preserve these mixtures. We study why through a combination of theoretical analysis and controlled empirical investigations on a variety of models. We identify three independent, distinct sources of failure. First, transformer architectures already distort mixture geometry, and training substantially amplifies this effect. Moreover, the failure can occur even if the model transports mixtures perfectly linearly: the softmax readout and autoregressive feedback form a dynamical system that either amplifies small differences until one component of the mixture dominates or contracts different mixtures until they become indistinguishable. We verify this theoretical prediction empirically: the observed transition between contraction and amplification occurs near the theoretical threshold derived by our analysis, and pretrained-model rollouts lie predominantly on the amplifying side. Finally, we generalize to mixtures of many components and show that exact preservation generally requires context-dependent correction, whose required dimensionality can grow with the number of components.
