As large language models (LLMs) scale, the question is not only how large they become, but how much of their capacity is effectively utilized. Existing scaling laws relate model size to loss, yet overlook how components exploit their latent space. We study feed-forward networks (FFNs) and recast width selection as a spectral utilization problem. Using a lightweight diagnostic suite -- Hard Rank (participation ratio), Soft Rank (Shannon rank), Spectral Concentration, and the composite Spectral Utilization Index (SUI) -- we quantify how many latent directions are meaningfully activated across LLaMA, GPT-2, and nGPT families. Our key finding is an asymmetric spectral scaling law: soft rank follows an almost perfect power law with FFN width, while hard rank grows only sublinearly and with high variance. This asymmetry suggests that widening FFNs mostly adds low-energy tail directions, while dominant-mode subspaces saturate early. Moreover, at larger widths, variance further collapses into a narrow subspace, leaving much of the latent space under-utilized. These results recast FFN width selection as a principled trade-off between tail capacity and dominant-mode capacity, offering concrete guidance for inference-efficient LLM design.
A Latent Variable Framework for Scaling Laws in Large Language Models
Department of Statistics, University of Michigan(密歇根大学统计系);IBM Research and CSAIL, MIT(IBM研究与麻省理工学院计算机科学与人工智能实验室);Institute of Foundation Models, MBZUAI(MBZUAI基础模型研究所)
A Bayesian latent class reinforcement learning framework to capture adaptive, feedback-driven travel behaviour
Choice Modelling Centre & Institute for Transport Studies, University of Leeds(选择建模中心及交通研究学院,利兹大学);DTU Management, Technical University of Denmark(技术大学丹麦管理学院);Institute of Transport and Logistics Studies, University of Sydney(交通与物流研究学院,悉尼大学);College of Business, Al Yamamah University, Riyadh, Saudi Arabia(商学院,阿尔亚玛马大学,利雅得,沙特阿拉伯)
Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families
Department of Statistics, University of Michigan(密歇根大学统计学系);MIT-IBM Watson AI Lab, IBM Research(MIT-IBM Watson AI实验室,IBM研究);Computer Science and Artificial Intelligence Laboratory, MIT(MIT计算机科学与人工智能实验室);Institute of Foundation Models, MBZUAI(基础模型研究所,MBZUAI)
Latent Adversarial Regularization for Offline Preference Optimization
Department of Computer Science, Stanford University(斯坦福大学计算机科学系);Siebel School of Computing(塞贝尔计算学院);Data Science, University of Illinois at Urbana-Champaign(数据科学,伊利诺伊大学厄巴纳-香槟分校)
National University of Singapore(新加坡国立大学);Center for AI Research, VinUniversity(Vin大学人工智能研究中心);Independent Researcher(独立研究者);Hanoi University of Science and Technology(河内科学技术大学)
tLaSDI: Thermodynamics-informed latent space dynamics identification
Korea Institute for Advanced Study (KIAS)(韩国高等研究院);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);North Carolina State University(北卡罗来纳州立大学);Pohang University of Science and Technology (POSTECH)(浦项科技大学)