Model dimension ($d_{model}$) is a fundamental hyperparameter in transformer language models, yet its role in setting the geometric limits of feature representation remains under-explored. Grounded in the Linear Representation and Superposition Hypotheses - which propose that models encode features as near-orthogonal directions in latent space - we develop a framework for estimating how many such directions a model can support. We first establish the embedding matrix as a measurable proxy for near-orthogonality constraints across the latent space: the boundary between meaningful token relationships and incidental similarity in the pairwise cosine similarity distribution gives a concrete estimate of the model's accepted deviation $\varepsilon$ from perfect orthogonality. Applying this metric across dozens of open-source models reveals two classes: models with high $\varepsilon$ whose embeddings lack near-orthogonal structure, and models with low $\varepsilon$ that maintain it. We then show that the standard Johnson-Lindenstrauss lemma greatly underestimates the packing efficiency of trained representations, and derive an adjusted capacity formula in which the number of near-orthogonal directions depends on the ratio of vectors to dimensions ($k/d$) rather than the raw count - a single modification that cuts prediction error by two orders of magnitude with no extra parameters. Combining these results, we define representational capacity as an upper bound on the number of distinguishable directions available for features and embeddings in a model's latent space. Capacity is exponentially sensitive to $\varepsilon$, and larger models favor tighter orthogonality constraints over maximizing raw capacity - a pattern compatible with several explanations (a stability-capacity trade-off, a ceiling on usable concepts, or confounds with model scale) that we leave to future work.
Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families
Department of Computer Science and Engineering, UC San Diego(计算机科学与工程系,UC San Diego);Department of Electrical and Computer Engineering, UC San Diego(电气与计算机工程系,UC San Diego);Department of Cognitive Science, UC San Diego(认知科学系,UC San Diego)
Why the Brain Consolidates: Predictive Forgetting for Optimal Generalisation
Huawei Noah’s Ark Lab(华为诺亚实验室);AI Centre, Department of Computer Science, University College London(大学计算机科学系人工智能中心);UCL Institute of Cognitive Neuroscience, University College London(大学认知神经科学研究所);UCL Queen Square Institute of Neurology, University College London(大学Queen Square神经学研究所)
Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
University of California San Diego(加州大学圣迭戈分校);University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Fermi National Accelerator Laboratory(费米国家加速器实验室);University of California Davis(加州大学戴维斯分校)
Geometric Capacity of Transformers: A Tropical Geometry Perspective
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院)
Localized, High-resolution Geographic Representations with Slepian Functions
Department of Computer Science, University of Colorado Boulder(科罗拉多大学波德分校计算机科学系);Microsoft(微软公司);University College London(伦敦大学学院);McGill University(麦吉尔大学);Mila–Quebec AI Institute(魁北克AI研究所);University of Bonn, Germany(德国波恩大学);University of Wageningen, Netherlands(荷兰瓦赫宁根大学)
Next-token prediction capacity: general upper bounds and a lower bound for transformers
Department of Computer Science, University of British Columbia(计算机科学系,不列颠哥伦比亚大学);Department of Electrical and Computer Engineering, University of British Columbia(电气与计算机工程系,不列颠哥伦比亚大学)