arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.28573cs.LGcs.AI
收藏

通过截断SVD层实现LLM的高效预训练

Efficient Pre-Training of LLMs through Truncated SVD Representations

Cognizant AI Lab(认知AI实验室);UT Austin(得克萨斯大学奥斯汀分校)

Kaivan Kamali, Kajetan Schweighofer, Hormoz Shahrzad, Olivier Francon, Babak Hodjat, Risto Miikkulainen

AI总结:

提出TSVD框架,利用谱能量启发式自适应秩选择和缓存机制保持低秩与严格正交性,在减少计算开销的同时匹配或超越全参数基线的性能。

摘要:

大规模语言模型(LLM)的规模扩展使得预训练成本日益高昂。虽然低秩表示和正交权重矩阵原则上可以减少参数数量和计算开销,但现有方法大多依赖静态秩选择,且由于高计算成本而不强制权重正交性。本文引入TSVD框架,在整个训练过程中保持低秩和严格正交性。它利用基于谱能量的启发式方法进行自适应秩选择,并采用缓存机制来维持正交性。理论分析证明了该方法在预训练动态中的优势,跨多种模型规模的实验表明其在经验上有效。TSVD在显著降低计算需求的同时,匹配或超越了全参数基线的性能。因此,该方法为高效高性能LLM预训练提供了一条有充分依据、实用且可扩展的路径。

英文摘要:

LLM pretraining is extremely costly; therefore, parameter-efficient LLM architectures have recently emerged as a compelling research direction. One such promising approach is to represent the parameters as orthonormal low-rank weight matrices. However, maintaining orthonormality during training is computationally expensive, making it impractical. This paper presents the TSVD (Truncated Singular Value Decomposition) framework which efficiently maintains orthonormality through QR decomposition and caching. Furthermore, a spectral energy heuristic is introduced to select the rank of the resulting low-rank weight matrices. Empirical evaluations across model sizes show that TSVD matches or outperforms full-parameter baselines at a fraction of the compute cost. TSVD thus provides a scalable, computationally efficient foundation for LLM pretraining.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑