arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.00952cs.LGcs.CLcs.DC
收藏

SplitLoRA:一种面向大语言模型的分割式参数高效微调框架

SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models

School of Computer Science, Fudan University(复旦大学计算机科学学院);University of Hong Kong(香港大学);University of Maryland(马里兰大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Massachusetts Institute of Technology(麻省理工学院)

Zheng Lin, Xuanjie Hu, Yuxin Zhang, Zhe Chen, Zihan Fang, Xianhao Chen, Ang Li, Praneeth Vepakomma, Yue Gao

AI总结:

本文提出首个分割学习大语言模型微调框架SplitLoRA,结合联邦学习并行训练与分割学习模型切分优势,在显著缩短训练时间的同时达到目标精度,并作为开源基准推动该范式研究。

摘要:

大语言模型(LLMs)在处理高复杂度模型和大规模数据集方面的可扩展性,已在关键领域取得了巨大成功。尽管获取更多LLM训练数据的需求十分迫切,但一个令人担忧的现实是,高质量公共数据集将在几年内耗尽。有鉴于此,联邦学习(FL)LLM微调范式最近被提出,以促进在分布式私有数据上的协作式LLM微调,其中多个数据所有者在不共享原始数据的情况下协作微调一个共享LLM。然而,LLM惊人的模型规模给客户端带来了沉重的计算和通信负担,对FL LLM微调范式的普及构成了重大障碍。为解决这一问题,分割学习(SL)作为一种有前景的解决方案应运而生,它通过模型划分将主要训练负载卸载到服务器,同时以较小的数据量交换激活值/激活值的梯度,而非整个LLM。遗憾的是,关于SL LLM微调范式的研究仍处于起步阶段。为填补这一空白,本文提出了首个SL LLM微调框架,命名为SplitLoRA。SplitLoRA构建于分割联邦学习(SFL)框架之上,融合了FL的并行训练优势和SL的模型分割优势,从而大幅提升了训练效率。值得注意的是,SplitLoRA是SL LLM微调的首个开源基准,为致力于推进SL LLM微调的研究工作提供了基础。大量仿真实验验证了SplitLoRA在显著少于最先进LLM微调框架的时间内达到目标精度,证明了SplitLoRA卓越的训练性能。项目页面见 https://fduinc.github.io/splitlora/。

英文摘要:

The scalability of large language models (LLMs) in handling high-complexity models and large-scale datasets has led to tremendous successes in pivotal domains. While there is an urgent need to acquire more training data for LLMs, a concerning reality is the depletion of high-quality public datasets within a few years. In view of this, the federated learning (FL) LLM fine-tuning paradigm recently has been proposed to facilitate collaborative LLM fine-tuning on distributed private data, where multiple data owners collaboratively fine-tune a shared LLM without sharing raw data. However, the staggering model size of LLMs imposes heavy computing and communication burdens on clients, posing significant barriers to the democratization of the FL LLM fine-tuning paradigm. To address this issue, split learning (SL) has emerged as a promising solution by offloading the primary training workload to a server via model partitioning while exchanging activation/activation's gradients with smaller data sizes rather than the entire LLM. Unfortunately, research on the SL LLM fine-tuning paradigm is still in its nascent stage. To fill this gap, in this paper, we propose the first SL LLM fine-tuning framework, named SplitLoRA. SplitLoRA is built on the split federated learning (SFL) framework, amalgamating the advantages of parallel training from FL and model splitting from SL and thus greatly enhancing the training efficiency. It is worth noting that SplitLoRA is the inaugural open-source benchmark for SL LLM fine-tuning, providing a foundation for research efforts dedicated to advancing SL LLM fine-tuning. Extensive simulations validate that SplitLoRA achieves target accuracy in significantly less time than state-of-the-art LLM fine-tuning frameworks, demonstrating the superior training performance of SplitLoRA. The project page is available at https://fduinc.github.io/splitlora/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑