Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies. Here, we quantify how depth affects loss via analysis of LLMs and toy residual networks. We find loss scales inversely proportional to depth in LLMs, probably due to functionally similar layers reducing error through ensemble averaging rather than compositional learning or discretizing smooth dynamics. This regime is inefficient yet robust and may arise from the architectural bias of residual networks and target functions incompatible with smooth dynamics. The findings suggest that improving LLM efficiency may require architectural innovations to encourage compositional use of depth.
Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Toronto（多伦多大学）；University of Technology Sydney（悉尼技术大学）；Google DeepMind（谷歌DeepMind）；University of Maryland, College Park（马里兰大学学院市分校）；Massachusetts Institute of Technology（麻省理工学院）
Uncovering Scaling Laws for Large Language Models via Inverse Problems
Singapore-MIT Alliance for Research and Technology（新加坡-麻省理工联盟研究技术中心）；Dept. of Computer Science, National University of Singapore（新加坡国立大学计算机科学系）；Agency for Science, Technology and Research（科技研究局）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究院）；SAP（SAP公司）；AI Singapore（AI新加坡）
FAR AI；New York University（纽约大学）；Stability AI；Google（谷歌）；Johns Hopkins University（约翰斯·霍普金斯大学）；Oxford University（牛津大学）；Boston University（波士顿大学）；Anthropic
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Gatsby Computational Neuroscience Unit, University College London（加布兹计算神经科学单位，伦敦大学学院）；Sainsbury Wellcome Centre, University College London（萨金斯·惠康中心，伦敦大学学院）
Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Baidu Inc（百度公司）
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；City University of Hong Kong（香港城市大学）
Variational quantum algorithms are the centerpiece of modern quantum programming. These algorithms involve training parameterized quantum circuits using a classical co-processor, an approach adapted partly from classical machine learning. An important subclass of these algorithms, designed for combinatorial optimization on currrent quantum hardware, is the quantum approximate optimization algorithm (QAOA). It is known that problem density - a problem constraint to variable ratio - induces under-parametrization in fixed depth QAOA. Density dependent performance has been reported in the literatu…
Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time
Center for Mathematical Sciences and Applications（数学科学与应用中心）；John A. Paulson School of Engineering and Applied Sciences（约翰·A·保罗森工程与应用科学学院）；Kempner Institute for the Study of Natural and Artificial Intelligence（自然与人工智能研究学院）；Harvard University（哈佛大学）
We show that the number of unique function mappings in a neural network hypothesis space is inversely proportional to $\prod_lU_l!$, where $U_{l}$ is the number of neurons in the hidden layer $l$.
Sorbonne Université（索邦大学）；MINES ParisTech（巴黎高等矿业学院）；PSL Research University（巴黎文理研究大学）；Institut Curie（居里研究所）；INSERM（法国国家健康与医学研究院）；Institut universitaire de France（法国大学研究院）；Google Research（谷歌研究院）；EPFL（洛桑联邦理工学院）；Califrais（加利弗雷）；Owkin（奥金公司）
Following the recent work on capacity allocation, we formulate the conjecture that the shattering problem in deep neural networks can only be avoided if the capacity propagation through layers has a non-degenerate continuous limit when the number of layers tends to infinity. This allows us to study a number of commonly used architectures and determine which scaling relations should be enforced in practice as the number of layers grows large. In particular, we recover the conditions of Xavier initialization in the multi-channel case, and we find that weights and biases should be scaled down as …
AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
Rutgers University（新泽西州立大学）；Nanyang Technological University（南洋理工大学）；University of Connecticut（康涅狄格大学）；Fudan University（复旦大学）；NVIDIA Research（NVIDIA研究）；Red Hat AI Innovation（红帽AI创新）；MIT-IBM Watson AI Lab（MIT-IBM沃森AI实验室）；Massachusetts Institute of Technology（麻省理工学院）
Fr. Conceicao Rodrigues Institute of Technology, Vashi（康塞桑·罗德里格斯理工学院，瓦希）
Convolutional Neural Networks can be designed with different levels of complexity depending upon the task at hand. This paper analyzes the effect of dimensional changes to the CNN architecture on its performance on the task of Histopathological Cancer Classification. The research starts with a baseline 10-layer CNN model with (3 X 3) convolution filters. Thereafter, the baseline architecture is scaled in multiple dimensions including width, depth, resolution and a combination of all of these. Width scaling involves inculcating greater number of neurons per CNN layer, whereas depth scaling invo…
Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
ELLIS Institute Tübingen（埃利斯研究所蒂宾根分所）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（蒂宾根人工智能中心）；University of Maryland, College Park（马里兰大学帕克分校）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Efficient Decoder Scaling Strategy for Neural Routing Solvers
Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory（广东省全驱动系统控制理论与技术重点实验室）；School of Automation（自动化学院）；Intelligent Manufacturing, Southern University of Science（智能制造，南方科技大学）；Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China（计算机科学系，香港城市大学，香港特别行政区，中国）
Breaking scaling relations with inverse catalysts: a machine learning exploration of trends in $\mathrm{CO_2}$ hydrogenation energy barriers
Center for Interstellar Catalysis, Department of Physics and Astronomy, Aarhus University, Aarhus C, Denmark（星际催化中心，物理与天文学系，奥胡斯大学，奥胡斯C，丹麦）；Aarhus Institute of Advanced Studies, Aarhus University, Aarhus C, Denmark（奥胡斯高级研究学院，奥胡斯大学，奥胡斯C，丹麦）
In this work we analyze strategies for convolutional neural network scaling; that is, the process of scaling a base convolutional network to endow it with greater computational complexity and consequently representational power. Example scaling strategies may include increasing model width, depth, resolution, etc. While various scaling strategies exist, their tradeoffs are not fully understood. Existing analysis typically focuses on the interplay of accuracy and flops (floating point operations). Yet, as we demonstrate, various scaling strategies affect model parameters, activations, and conse…
On the Depth Transfer of Neural Optimizers via L-Smoothness
University of Chicago（芝加哥大学）；Peking University（北京大学）；University of British Columbia（不列颠哥伦比亚大学）；Sauder School of Business（尚德商学院）；Beijing International Center for Mathematical Research（北京国际数学研究中心）
