发表机构
Electronics and Telecommunications Research Institute（电子和电信研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究大语言模型中Transformer动力学集体行为的物理机制，利用Pythia语言模型提取弛豫谱，定量测量相关指标，发现其动力学受红外集体组织支配，红外慢模式组织是普遍集体原则。
AI 中文摘要
大语言模型展现出显著的涌现行为，但其集体动力学的物理机制仍知之甚少。认知场理论预测，学习通过慢弛豫模式的红外积累来重组状态的时间尺度密度（TDOS），从而增强记忆自能、减小认知遗忘差距并增强集体敏感性。利用公开可用的Pythia语言模型，我们在整个训练、网络深度和模型规模中直接从Transformer层雅可比矩阵提取弛豫谱，从而能够定量测量TDOS、记忆自能、遗忘差距、记忆核和红外临界指数。测量结果揭示了慢弛豫模式的渐进红外积累，产生了近似平坦的红外TDOS，其\(\rho(\lambda)\sim\lambda^{-0.1}\)，以及无标度记忆核\(K(t)\sim t^{-1}\)。记忆自能在早期优化期间表现出明显的瞬态最大值，然后朝着亚稳态近临界状态弛豫，这对应于认知场理论预测的最小认知遗忘差距和最大集体敏感性。这些观察结果提供了定量实验证据，表明Transformer动力学受红外集体组织支配。相同动力学行为在训练、网络深度和模型规模上的可重复性表明，红外慢模式组织代表了Transformer动力学的普遍集体原则。
英文摘要
Large language models exhibit remarkable emergent behaviors, yet the physical mechanism governing their collective dynamics remains poorly understood. Cognitive Field Theory predicts that learning reorganizes the collective relaxation spectrum, thereby modifying memory self-energy, long-memory dynamics, and collective susceptibility through the infrared organization of slow relaxation modes. Here we test this framework directly in Transformer dynamics. Using publicly available Pythia language models, we extract relaxation spectra from layer Jacobians throughout training, prompt ensembles, network depth, and model scale, allowing the collective observables of Cognitive Field Theory to be measured quantitatively. The measurements reveal pronounced infrared reorganization of the relaxation spectrum. Learning substantially redistributes spectral weight while preserving a nearly flat but weakly infrared-enhanced time-scale density of states, \( ρ(λ)\simλ^β, \qquad β\simeq-0.1, \) with a corresponding memory kernel exhibiting robust long-memory scaling close to \( K(t)\sim\frac{1}{t}. \) The collective observables further reveal a critical formation process: the memory self-energy reaches a transient maximum during early training before relaxing toward a metastable near-critical regime. Prompt-resolved and token-subspace measurements show that distinct local Jacobians recover a common macroscopic TDOS with shared infrared scaling, consistent with infrared fixed-point organization under coarse graining. The reproducibility of this infrared organization across training, prompt ensembles, network depth, and Transformer model scales supports infrared slow-mode organization as a robust collective principle of Transformer dynamics, providing a quantitative experimental realization of the collective observables predicted by Cognitive Field Theory.
Comments55 pages, 47 figures
