arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.08727cs.LG
收藏

基于张量分解的变换低秩适配及其在文本生成图像模型中的应用

Transformed Low-rank Adaptation via Tensor Decomposition and Its Applications to Text-to-image Models

RIKEN AIP(RIKEN人工智能研究所);Sony AI(索尼人工智能);Sony Group Corporation(索尼集团)

Zerui Tao, Yuhta Takida, Naoki Murata, Qibin Zhao, Yuki Mitsufuji

AI总结:

针对LoRA低秩假设与微调权重的近似差距问题,提出融合变换与残差适配的张量分解PEFT方法,在Stable Diffusion微调任务上实现了更优的性能与参数效率。

摘要:

文本生成图像模型的参数高效微调(Parameter-Efficient Fine-Tuning, PEFT)已成为应用日益广泛的热门技术。在各类PEFT方法中,低秩适配(Low-Rank Adaptation, LoRA)及其变体凭借有效性获得了广泛关注,让用户能够在计算资源有限的情况下完成模型微调。然而,低秩假设与目标微调权重之间存在近似差距,导致无法同时实现超参数高效性与更优性能。为缩小这一差距并进一步提升LoRA的性能,我们提出了一种新型PEFT方法,该方法融合了两类适配机制:变换适配与残差适配。具体而言,我们首先对预训练权重施加一个全秩稠密变换,该可学习变换旨在让预训练权重尽可能贴近目标权重,从而降低残差权重的秩。随后,残差部分可通过更紧凑、参数更高效的结构实现有效近似,且近似误差更小。为在实际应用中实现超参数高效性,我们为变换适配和残差适配都设计了高度灵活且有效的张量分解方案。此外,DoRA等主流PEFT方法都可以纳入这种“变换+残差适配”的框架之下。我们在Stable Diffusion模型的主体驱动生成与可控生成微调任务上开展了实验,结果表明,与LoRA及多个基线方法相比,本文方法能够实现更优的性能与参数效率。

英文摘要:

Parameter-Efficient Fine-Tuning (PEFT) of text-to-image models has become an increasingly popular technique with many applications. Among the various PEFT methods, Low-Rank Adaptation (LoRA) and its variants have gained significant attention due to their effectiveness, enabling users to fine-tune models with limited computational resources. However, the approximation gap between the low-rank assumption and desired fine-tuning weights prevents the simultaneous acquisition of ultra-parameter-efficiency and better performance. To reduce this gap and further improve the power of LoRA, we propose a new PEFT method that combines two classes of adaptations, namely, transform and residual adaptations. In specific, we first apply a full-rank and dense transform to the pre-trained weight. This learnable transform is expected to align the pre-trained weight as closely as possible to the desired weight, thereby reducing the rank of the residual weight. Then, the residual part can be effectively approximated by more compact and parameter-efficient structures, with a smaller approximation error. To achieve ultra-parameter-efficiency in practice, we design highly flexible and effective tensor decompositions for both the transform and residual adaptations. Additionally, popular PEFT methods such as DoRA can be summarized under this transform plus residual adaptation scheme. Experiments are conducted on fine-tuning Stable Diffusion models in subject-driven and controllable generation. The results manifest that our method can achieve better performances and parameter efficiency compared to LoRA and several baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑