arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.23901cs.LGcs.AIcs.ITmath.IT
收藏

LLMs 作为噪声信道:香农视角下的模型容量与缩放定律

LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

University of Virginia(弗吉尼亚大学);University of California, Berkeley(加州大学伯克利分校)

Xu Ouyang, Deyi Liu, Yuhang Cai, Jing Liu, Yuan Yang, Chen Zheng, Thomas Hartvigsen, Yiyuan Ma

AI总结:

针对现有缩放定律无法解释非单调性能退化的问题,提出基于香农-哈特利定理的香农缩放定律,将LLM训练建模为噪声信道上的信息传输,通过信号噪声比解释模型规模与数据量对性能的影响。

摘要:

现有的大语言模型(LLMs)缩放定律主要是单调幂律,无法解释新出现的非单调现象,如灾难性过训练和量化引起的退化,在这些现象中,尽管计算量增加,性能却下降。我们提出了香农缩放定律,这是一个统一的理论框架,将LLM训练建模为噪声信道上的信息传输,基于香农-哈特利定理。通过将模型参数映射到信道带宽,训练令牌映射到信号功率,我们的公式明确捕捉了学习信号与内在噪声之间的相互作用。这一视角揭示了LLMs的基本香农容量:在未保持足够信噪比(SNR)的情况下扩展模型规模或数据,必然会放大噪声,导致从单调改进到U形性能退化的转变。我们通过在Pythia和OLMo2上进行的实验验证了该理论,实验包括高斯噪声、量化以及在数学、问答和代码任务上的监督微调。香农缩放定律始终优于经典缩放定律和最近的扰动感知定律,取得了强$R^2$分数,并准确捕捉了先前方法遗漏的损失盆地。它还能进行外推:在$\leq$6.9B Pythia模型上使用$\leq$180B令牌拟合后,预测了未见过的12B模型在高达307B令牌时的性能,池化$R^2=0.847$,而单调基线则崩溃。

英文摘要:

Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, our formulation explicitly captures the interaction between learning signal and intrinsic noise. This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, inducing a transition from monotonic improvement to U-shaped performance degradation. We validate our theory through experiments on Pythia and OLMo2 under perturbations, including Gaussian noise, quantization and supervised fine-tuning on math, QA and code tasks. The Shannon Scaling Law consistently outperforms classical scaling laws and recent perturbation-aware laws, achieving strong $R^2$ scores and accurately capturing loss basins missed by prior approaches. It also extrapolates: fitted on $\leq$6.9B Pythia models with $\leq$180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled $R^2{=}0.847$, while monotonic baselines collapse.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑