Pre-training large language models is dominated by the memory cost of storing full-rank weights, gradients, and optimizer states. Low-rank pre-training has emerged to address this, and the space of methods has grown rapidly. A central question remains open: do low-rank methods produce models that generalize comparably to full-rank training, or does the rank constraint fundamentally alter the solutions reached? Existing comparisons rely almost entirely on validation perplexity from single-seed runs, often carried forward from prior literature. Yet perplexity is a poor proxy for solution quality; two methods can match on perplexity while converging to different loss landscape regions and internal representations. We close this gap by characterizing the solutions found by five low-rank pre-training methods, GaLore and Fira (memory-efficient optimizers), CoLA and SLTrain (architecture reparameterizations), and ReLoRA (adapter-style updates with periodic resets), against full-rank training at three model scales (60M, 130M, 350M). We evaluate each along 16 metrics across four dimensions: 1-D loss landscape along random/top-K PCA directions, 1-D interpolation between checkpoints, spectral structure of the weights and learned updates, and activation similarity to full-rank training. We show that low-rank methods are not equivalent to full-rank training, nor to one another, even when validation perplexity is close. Full-rank training settles into a sharper basin than low-rank methods along random directions, while the reverse holds for the top-1 PCA direction. Each method converges to a geometrically distinct basin. Low-rank activations diverge from full-rank in later layers as training progresses, with GaLore tracking full-rank most closely. Further, validation perplexity does not translate to downstream performance at every scale. Adding geometric and spectral metrics improves the prediction.
Pazhou Laboratory(琶洲实验室);Zhejiang University(浙江大学);South China Agricultural University(华南农业大学);Chongqing University of Posts and Telecommunications(重庆邮电大学);Key Laboratory of Big Data and Intelligent Robot, Ministry of Education(教育部大数据与智能机器人重点实验室)
Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
Department of Computer Science, Yonsei University(yonsei大学计算机科学系);Department of Artificial Intelligence, Yonsei University(yonsei大学人工智能系);Department of Electronic & Electrical Convergence Engineering, Hongik University(hongik大学电子与电气融合工程系)
UniRank: Unified Rank Allocation for Low-Rank LLM Compression
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo(宁波数字孪生研究院,东方理工大学(宁波));Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳))
SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学);Carnegie Mellon University(卡内基梅隆大学)
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
Department of Computer Science, Aalborg University Copenhagen(奥尔堡大学哥本哈根分校计算机科学系);MaLGa-DIBRIS, University of Genoa(热那亚大学MaLGa-DIBRIS);INFN, Sezione di Genova(国家核物理研究所热那亚分部);European Organization for Nuclear Research (CERN)(欧洲核子研究中心);Ceva, Inc.(Ceva公司)
On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
Carnegie Mellon University(卡内基梅隆大学);National Taiwan University(国立台湾大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所);Massachusetts Institute of Technology(麻省理工学院)
Bridging Internal Probability and Self-Consistency for Effective and Efficient LLM Reasoning
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)
Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity
College of Informatics, University of Amsterdam(阿姆斯特丹大学信息学学院);Department of Medical Informatics, Amsterdam University Medical Center(阿姆斯特丹大学医学中心医学信息学系);ILLC, University of Amsterdam(阿姆斯特丹大学逻辑、语言与计算研究所)