Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute.
We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, our formulation explicitly captures the interaction between learning signal and intrinsic noise. This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, inducing a transition from monotonic improvement to U-shaped performance degradation.
We validate our theory through experiments on Pythia and OLMo2 under perturbations, including Gaussian noise, quantization and supervised fine-tuning on math, QA and code tasks. The Shannon Scaling Law consistently outperforms classical scaling laws and recent perturbation-aware laws, achieving strong $R^2$ scores and accurately capturing loss basins missed by prior approaches. It also extrapolates: fitted on $\leq$6.9B Pythia models with $\leq$180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled $R^2{=}0.847$, while monotonic baselines collapse.
Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws
Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);School of Computer Science and Engineering, Beihang University(北航计算机科学与工程学院)
Symbolic Graph Networks for Robust PDE Discovery from Noisy Sparse Data
School of Software Engineering, Chengdu University of Information Technology(成都信息科技学院软件工程学院);School of Statistics, Chengdu University of Information Technology(成都信息科技学院统计学院);College of Applied Mathematics, Chengdu University of Information Technology(成都信息科技学院应用数学学院);Key Laboratory of Mathematical Meteorology, Chengdu University of Information Technology(成都信息科技学院数学气象重点实验室)
Reranking Laws for Language Generation: A Communication-Theoretic Perspective
Instituto Superior Técnico, Universidade de Lisboa(里斯本大学高等技术学院);Instituto de Telecomunicações(电信研究所);TU Darmstadt(达姆施塔特工业大学);ELLIS Unit Lisbon(欧洲卓越学习系统里斯本单元);Unbabel(昂巴贝尔公司)
Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error
Principles of Informatics Research Division, National Institute of Informatics (NII)(国立信息学研究所信息学原理研究部);Informatics Program, Graduate Institute for Advanced Studies (SOKENDAI)(综合研究大学院大学信息学项目)
Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning
Department of Electrical and Computer Engineering(电气与计算机工程系);Georgia Institute of Technology(佐治亚理工学院);Department of Mathematics(数学系);Massachusetts Institute of Technology(麻省理工学院)