arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.13652cs.LGcs.AIcs.CL
收藏

超越困惑度:低秩预训练的几何与谱研究

Beyond Perplexity: A Geometric and Spectral Study of Low-Rank Pre-Training

University of Massachusetts Lowell(马萨诸塞大学洛厄尔分校)

Namrata Shivagunde, Vijeta Deshpande, Sherin Muckatira, Anna Rumshisky

AI总结:

本文通过几何和谱分析研究低秩预训练方法,揭示其与全秩训练在模型性能和解空间上的差异,发现低秩方法在不同模型规模下表现各异,且困惑度不能完全反映下游任务性能。

摘要:

大规模语言模型的预训练主要受限于存储全秩权重、梯度和优化器状态的内存成本。低秩预训练出现以解决这一问题,相关方法空间迅速扩展。一个核心问题仍未解决:低秩方法是否能产生与全秩训练具有同等泛化能力的模型,或者秩约束是否根本性地改变了所达到的解?现有比较几乎完全依赖于单种子运行的验证困惑度,通常继承自先前文献。然而,困惑度是解质量的差代理;两种方法可以在困惑度上匹配,却收敛到不同的损失景观区域和内部表示。我们通过表征五种低秩预训练方法(GaLore和Fira(内存高效优化器)、CoLA和SLTrain(架构再参数化)、ReLoRA(适配器式更新带周期性重置))在三个模型规模(60M、130M、350M)下与全秩训练的解,关闭这一差距。我们评估每种方法在四个维度上的16个指标:1D损失景观沿随机/Top-K PCA方向、1D检查点之间插值、权重和学习更新的谱结构,以及激活相似性与全秩训练。我们显示低秩方法不等同于全秩训练,也不等同于彼此,即使验证困惑度接近。全秩训练在随机方向上达到更尖锐的盆地,而反方向则适用于top-1 PCA方向。每种方法收敛到几何上不同的盆地。低秩激活在训练过程中随着层数增加而偏离全秩激活,GaLore最接近全秩激活。进一步,验证困惑度在每个规模下并不转化为下游性能。添加几何和谱度量提高了预测。

英文摘要:

Pre-training large language models is dominated by the memory cost of storing full-rank weights, gradients, and optimizer states. Low-rank pre-training has emerged to address this, and the space of methods has grown rapidly. A central question remains open: do low-rank methods produce models that generalize comparably to full-rank training, or does the rank constraint fundamentally alter the solutions reached? Existing comparisons rely almost entirely on validation perplexity from single-seed runs, often carried forward from prior literature. Yet perplexity is a poor proxy for solution quality; two methods can match on perplexity while converging to different loss landscape regions and internal representations. We close this gap by characterizing the solutions found by five low-rank pre-training methods, GaLore and Fira (memory-efficient optimizers), CoLA and SLTrain (architecture reparameterizations), and ReLoRA (adapter-style updates with periodic resets), against full-rank training at three model scales (60M, 130M, 350M). We evaluate each along 16 metrics across four dimensions: 1-D loss landscape along random/top-K PCA directions, 1-D interpolation between checkpoints, spectral structure of the weights and learned updates, and activation similarity to full-rank training. We show that low-rank methods are not equivalent to full-rank training, nor to one another, even when validation perplexity is close. Full-rank training settles into a sharper basin than low-rank methods along random directions, while the reverse holds for the top-1 PCA direction. Each method converges to a geometrically distinct basin. Low-rank activations diverge from full-rank in later layers as training progresses, with GaLore tracking full-rank most closely. Further, validation perplexity does not translate to downstream performance at every scale. Adding geometric and spectral metrics improves the prediction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑