We investigate how different compression techniques -- such as weight and activation quantization, and weight sparsity -- affect the scaling behavior of large language models (LLMs) during pretraining. Building on previous work showing that weight sparsity acts as a constant multiplier on model size in scaling laws, we demonstrate that this "effective parameter" scaling pattern extends to quantization as well. Specifically, we establish that weight-only quantization achieves strong parameter efficiency multipliers, while full quantization of both weights and activations shows diminishing returns at lower bitwidths. Our results suggest that different compression techniques can be unified under a common scaling law framework, enabling principled comparison and combination of these methods.
Many networked systems such as electric networks, the brain, and social networks of opinion dynamics are known to obey conservation laws. Examples of this phenomenon include the Kirchoff laws in electric networks and opinion consensus in social networks. Conservation laws in networked systems may be modeled as balance equations of the form $X = B^{*} Y$, where the sparsity pattern of $B^{*}$ captures the connectivity of the network, and $Y, X \in \mathbb{R}^p$ are vectors of "potentials" and "injected flows" at the nodes respectively. The node potentials $Y$ cause flows across edges and the fl…
Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules
Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学多媒体信息处理国家重点实验室）；AI for Science Institute, Beijing（北京人工智能科学研究院）
ARC Training Centre for Information Resilience (CIRES)（信息韧性ARC培训中心（CIRES））；School of Mathematics and Physics, University of Queensland（昆士兰大学数学与物理学院）；Institute for Molecular Bioscience, University of Queensland（昆士兰大学分子生物科学研究所）；Profenso（Profenso公司）；QIMR Berghofer Medical Research Institute（QIMR伯格霍尔医学研究中心）
