arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.12429cs.LG
收藏

稳定原生低秩大语言模型预训练

Stabilizing Native Low-Rank LLM Pretraining

Concordia University, Montreal, Canada(Concordia 大学);Mila Quebec AI Institute, Montreal, Canada(Quebec AI 研究院);Sorbonne University, CNRS , Paris , France(Sorbonne 大学)

Paul Janson, Edouard Oyallon, Eugene Belilovsky

AI总结:

本文提出Spectron方法,通过谱归一化实现稳定原生低秩大语言模型预训练,显著提升推理效率并优化计算资源使用。

摘要:

基础模型已取得显著成功,但其快速增长的参数数量带来了显著的计算和内存挑战。低秩分解提供了一条减少训练和推理成本的有希望的途径,但社区缺乏一种稳定的配方,用于仅使用低秩权重从头开始训练模型,同时匹配密集模型的性能。我们证明,大型语言模型(LLMs)可以仅使用低秩分解权重对所有非嵌入矩阵进行从头开始的训练,而无需先前方法所需的辅助

英文摘要:

Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promising route to reduce training and inference costs, but the community lacks a stable recipe for training models from scratch using exclusively low-rank weights while matching the performance of the dense model. We demonstrate that Large Language Models (LLMs) can be trained from scratch using exclusively low-rank factorized weights for all non-embedding matrices without auxiliary "full-rank" guidance required by prior methods. While native low-rank training often suffers from instability and loss spikes, we identify uncontrolled growth in the spectral norm (largest singular value) of the weight matrix update as the dominant factor. To address this, we introduce Spectron: Spectral renormalization with orthogonalization, which dynamically bounds the resultant weight updates based on the current spectral norms of the factors. Our method enables stable, end-to-end factorized training with negligible overhead. Finally, we establish compute-optimal scaling laws for natively low-rank transformers, demonstrating predictable power-law behavior and improved inference efficiency relative to dense models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑