大型语言模型中跨深度的涌现因果几何动力学
Emergent Causal-Geometric Dynamics Across Depth in Large Language Models
AI总结：
通过结合几何分析与因果干预，揭示了解码器-only大型语言模型中从上下文处理到预测形成的跨层转变，并发现后期层中角度结构参数化下一词分布相似性并实现选择性因果控制。
摘要：
对大型语言模型（LLM）表征的几何分析揭示了跨深度的结构化变化，但本质上与token预测形成相关。同时，因果干预揭示了依赖于深度的效能曲线，但缺乏对其表征动力学的统一解释。对LLM功能的完整解释需要说明表征结构如何跨深度演化以因果性地产生预测。我们通过将几何分析与机械干预相结合，明确将跨深度动力学作为解释LLM功能的组织轴，综合了这些视角。在解码器-only LLM中，我们识别出从上下文处理到预测形成计算的急剧转变，伴随着跨层的表征几何的更渐进重组。这种综合揭示了一种后期层几何编码，其中角度结构参数化下一词分布相似性，并能够对预测进行选择性因果控制，而表征范数编码的信息与预测基本解耦。总之，我们的结果提供了因果和几何视角的综合，产生了关于语言模型中跨深度的控制相关几何动力学如何将上下文转化为预测的机械论解释。这一视角调和了先前令人困惑的发现，并表明层状功能不能孤立地理解或有效干预，而只能在网络涌现的全局动力学结构中理解。
英文摘要：
Geometric analyses of large language model (LLM) representations reveal structured variation across depth but remain fundamentally correlational with respect to token prediction formation. Meanwhile, causal interventions expose depth-dependent efficacy profiles without a unifying account of their representational dynamics. A complete account of LLM function requires explaining how representational structure evolves across depth to causally produce predictions. We synthesize these perspectives by combining geometric analysis with mechanistic interventions, explicitly centralizing depth-wise dynamics as the organizing axis for interpreting LLM function. In decoder-only LLMs, we identify a sharp transition from context-processing to prediction-forming computation, accompanied by a more gradual reorganization of representational geometry across layers. This synthesis reveals a late-layer geometric code in which angular structure parameterizes next-token distributional similarity and enables selective causal control over predictions, while representation norms encode information largely decoupled from prediction. Together, our results provide a synthesis of causal and geometric perspectives, yielding a mechanistic account of how control-relevant geometric dynamics across depth transform context into prediction in language models. This perspective reconciles previously puzzling findings and implies that layer-wise function cannot be understood or effectively intervened upon in isolation, but only within the emergent global dynamical structure of the network.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
