发表机构
College of Computing and Data Science; Nanyang Technological University（计算与数据科学学院; 南洋理工大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出SBD框架，将系统行为作为贝叶斯特征纳入ELBO，揭示因果性存在“因果税”现象，并构建非因果变分族GSH，通过NTK谱验证其能降低误差、提升泛化，为LM基础模型设计开辟新方向。
AI 中文摘要
我们证明语言模型（LMs）的因果性可能既非必要也非最优。当系统行为（记为$S$）作为第一性原理贝叶斯特征被纳入时，情况即是如此。此处，$S$指数据空间之外的额外主导因素，且涉及耦合效应。尽管因果性是现代架构的事实基础，近期研究却表明其与因果性存在持续的不匹配和矛盾。这些问题主要源于系统行为而非数据分布。因此，我们提出SBD框架，将$S$作为证据下界（ELBO）的不可约组成部分。SBD在理论上揭示了一种反直觉的“因果税”现象，即由于对$S$的忽视，因果性成为一种带有额外结构误差的次优近似。为应对潜变量分析的挑战，我们通过隐式测量、理论界引导的控制以及神经正切核（NTK）评估来验证SBD预测的$S$的影响。特别地，我们构建了Green Shell（GSH）以展示降低因果税的可能性。GSH是一个非因果变分族，它将$S$组件的顺序依赖链替换为分而治之的划分。在惰性训练机制下的NTK谱确认，GSH始终比因果性实现更紧的误差界，信噪比提升$7dB+$。在惰性训练的较后期阶段，GSH进一步带来更优的泛化能力（多尺度拟合能力丰富高达$20\%$）。综上，SBD将系统行为确立为因果性和分布拟合之外的一种互补理论抽象，为设计和优化LM基础模型开辟了新的研究途径。
英文摘要
We show that the causality of language models (LMs) may not be necessary nor optimal. This is the case when system behavior (denoted as $S$) is incorporated as a first-principle Bayesian feature. Here, $S$ refers to extra dominant factors beyond the data space, and they involve coupled effects. Despite being the de facto foundation of modern architecture, recent studies indicate persistent mismatches and contradictions with causality. These issues largely stem from system behavior rather than the data distribution. We therefore propose the SBD framework, which incorporates $S$ as an irreducible component of the evidence lower bound (ELBO). SBD theoretically reveals a counter-intuitive Causality Tax phenomenon, where causality emerges as a suboptimal approximation with an additional structural error, due to the obliviousness to $S$. To address the challenge of latent variable analysis, we validate the SBD-predicted impact of $S$ via implicit measurements, theoretical-bound-guided controls, and Neural Tangent Kernel (NTK) evaluations. In particular, we construct Green Shell (GSH) to show the possibility of reducing Causality Tax. GSH is a non-causal variational family, and it replaces the sequential dependency chain of $S$ components with a divide-and-conquer partition. NTK spectra in the lazy-training regime confirm that GSH always achieves significantly tighter error bounds than causality, with $7dB+$ improvement in signal-to-noise ratio. In the relatively later stage of lazy-training, GSH further leads to superior generalization (up to $20\%$ richer multi-scale fitting capabilities). Taken together, SBD establishes system behavior as a complementary theoretical abstraction besides causality and distribution fitting, opening new research avenues such as designing and optimizing LM base models.
