arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.06243cs.LGcs.AIcs.CLcs.CV
收藏

通过 Butterfly 分解实现参数高效的正交微调

Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization

Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Cambridge(剑桥大学);ETH Zürich(苏黎世联邦理工学院);University of Tübingen(蒂宾根大学);Mila(米拉研究所);Université de Montréal(蒙特利尔大学);The Alan Turing Institute(艾伦·图灵研究所)
Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Wel… 展开作者

Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, Bernhard Schölkopf

AI总结:

本文提出了一种基于 Butterfly 结构的正交微调方法 BOFT,通过高效的正交参数化大幅减少可训练参数,并在视觉和语言下游任务中验证了其有效性。

摘要:

大型基础模型正变得无处不在,但从头训练它们的成本极其高昂。因此,高效地将这些强大的模型适配到下游任务变得日益重要。在本文中,我们研究了一种有原则的微调范式——正交微调(Orthogonal Finetuning, OFT)——用于下游任务适配。尽管展现出良好的泛化能力,但由于正交矩阵的高维度,OFT 仍然使用了相当多的可训练参数。为了解决这个问题,我们首先从信息传输的角度审视 OFT,然后识别出几个能够实现更好参数效率的关键期望。受 Cooley-Tukey 快速傅里叶变换算法如何实现高效信息传输的启发,我们提出了一种使用 butterfly 结构的高效正交参数化方法。我们将这种参数化应用于 OFT,创建了一种新颖的参数高效微调方法,称为 Orthogonal Butterfly (BOFT)。通过将 OFT 作为特例纳入,BOFT 引入了一个广义的正交微调框架。最后,我们进行了广泛的实证研究,将大型视觉 Transformer、大型语言模型和文本到图像扩散模型适配到视觉和语言的各种下游任务中。

英文摘要:

Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from an information transmission perspective, and then identify a few key desiderata that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast Fourier transform algorithm enables efficient information transmission, we propose an efficient orthogonal parameterization using butterfly structures. We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. Finally, we conduct an extensive empirical study of adapting large vision transformers, large language models, and text-to-image diffusion models to various downstream tasks in vision and language.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑