arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01775cs.CL
收藏

OLoRA:大语言模型的正交低秩适配

OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models

Trylon AI(特赖隆人工智能公司)

Kerim Büyükakyüz

AI总结:

本文提出OLoRA方法,通过QR分解进行正交矩阵初始化来改进LoRA,在保持参数效率和GPU内存优势的同时显著加速大语言模型微调收敛并提升性能。

摘要:

大语言模型(LLMs)的出现彻底变革了自然语言处理,使模型在理解和生成类人文本方面具备了前所未有的能力。然而,与微调这些模型相关的计算成本和收敛时间仍然是重大挑战。低秩适配(LoRA)通过引入具有更少可训练参数的高效微调技术,已成为缓解这些问题的有前景的方法。在本文中,我们提出了OLoRA,这是对LoRA方法的一种增强,它通过QR分解利用正交矩阵初始化。OLoRA显著加速了LLM训练的收敛,同时保持了LoRA的效率优势,例如可训练参数数量和GPU内存占用。我们的实证评估表明,与标准LoRA相比,OLoRA不仅收敛更快,而且在各种语言建模任务上表现出更优的性能。这一进展为更高效、更易获取的LLM微调开辟了新途径,有望促进自然语言应用中的更广泛采用和创新。

英文摘要:

The advent of large language models (LLMs) has revolutionized natural language processing, enabling unprecedented capabilities in understanding and generating human-like text. However, the computational cost and convergence times associated with fine-tuning these models remain significant challenges. Low-Rank Adaptation (LoRA) has emerged as a promising method to mitigate these issues by introducing efficient fine-tuning techniques with a reduced number of trainable parameters. In this paper, we present OLoRA, an enhancement to the LoRA method that leverages orthonormal matrix initialization through QR decomposition. OLoRA significantly accelerates the convergence of LLM training while preserving the efficiency benefits of LoRA, such as the number of trainable parameters and GPU memory footprint. Our empirical evaluations demonstrate that OLoRA not only converges faster but also exhibits improved performance compared to standard LoRA across a variety of language modeling tasks. This advancement opens new avenues for more efficient and accessible fine-tuning of LLMs, potentially enabling broader adoption and innovation in natural language applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑