arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.13844cs.LGcs.AI
收藏

ColA:基于梯度学习的协同适应

ColA: Collaborative Adaptation with Gradient Learning

Duke University(杜克大学);University of Minnesota-Twin Cities(明尼苏达大学双城分校)

Enmao Diao, Qi Le, Suya Wu, Xinran Wang, Ali Anwar, Jie Ding, Vahid Tarokh

AI总结:

提出无参数微调方法ColA,通过梯度学习解耦隐藏表示与参数的梯度计算并卸载至低成本设备,在降低FTaaS计算开销的同时,于多基准上表现与现有PEFT方法相当或更优。

摘要:

反向传播的一个主要功能是计算隐藏表示和参数的梯度,以通过梯度下降进行优化。由于参数规模庞大,训练大型模型需要高昂的计算成本。尽管参数高效微调(PEFT)方法旨在训练较小的辅助模型以节省计算空间,但它们仍存在计算开销,特别是在面向众多用户的微调即服务(FTaaS)场景中。我们提出了基于梯度学习(GL)的协同适应(ColA),这是一种无参数、与模型无关的微调方法,它将隐藏表示梯度和参数梯度的计算解耦。与PEFT方法相比,ColA通过将梯度计算卸载到低成本设备上,促进了更具成本效益的FTaaS。我们还对ColA进行了理论分析,并实验证明ColA在各种基准上的表现与现有PEFT方法相当或更好。

英文摘要:

A primary function of back-propagation is to compute both the gradient of hidden representations and parameters for optimization with gradient descent. Training large models requires high computational costs due to their vast parameter sizes. While Parameter-Efficient Fine-Tuning (PEFT) methods aim to train smaller auxiliary models to save computational space, they still present computational overheads, especially in Fine-Tuning as a Service (FTaaS) for numerous users. We introduce Collaborative Adaptation (ColA) with Gradient Learning (GL), a parameter-free, model-agnostic fine-tuning approach that decouples the computation of the gradient of hidden representations and parameters. In comparison to PEFT methods, ColA facilitates more cost-effective FTaaS by offloading the computation of the gradient to low-cost devices. We also provide a theoretical analysis of ColA and experimentally demonstrate that ColA can perform on par or better than existing PEFT methods on various benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑