arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.07856cs.CVcs.AI
收藏

DTL:面向视觉识别的解耦迁移学习

DTL: Disentangled Transfer Learning for Visual Recognition

Nanjing University(南京大学)

Minghao Fu, Ke Zhu, Jianxin Wu

AI总结:

针对现有参数高效迁移学习方法可训练参数与骨干网络耦合导致GPU显存占用未有效降低的问题,提出基于轻量紧凑侧网络的解耦迁移学习DTL,在减少显存和参数的同时提升了准确率,刷新多个基准最优性能。

摘要:

当预训练模型规模快速增大时,在下游任务上进行微调的成本也在稳步上升。为了经济地微调这些模型,研究者提出了参数高效迁移学习(PETL,parameter-efficient transfer learning)方法,这类方法仅调整极小一部分可训练参数,就能高效学习高质量表征。然而,当前的PETL方法面临着一个困境:训练过程中GPU显存占用并未随可训练参数减少而有效降低。如果全量微调会遇到GPU显存不足的问题,PETL同样可能失败。出现这一现象的原因是,这些方法的可训练参数通常与骨干网络耦合在一起,因此必须在GPU显存中存储大量中间状态以用于梯度传播。为缓解这一问题,我们提出了解耦迁移学习(DTL,Disentangled Transfer Learning),该方法利用轻量级的紧凑侧网络(CSN,Compact Side Network)将可训练参数与骨干网络解耦。通过少量低秩线性映射逐步提取任务特定信息,并将该信息适当地加回骨干网络,CSN在各类下游任务中有效实现了知识迁移。我们开展了大量实验以验证该方法的有效性。所提方法不仅大幅降低了GPU显存占用量和可训练参数量,还在准确率上显著优于现有PETL方法,在多个标准基准上取得了新的最优性能。代码可在https://github.com/heekhero/DTL获取。

英文摘要:

When pre-trained models become rapidly larger, the cost of fine-tuning on downstream tasks steadily increases, too. To economically fine-tune these models, parameter-efficient transfer learning (PETL) is proposed, which only tunes a tiny subset of trainable parameters to efficiently learn quality representations. However, current PETL methods are facing the dilemma that during training the GPU memory footprint is not effectively reduced as trainable parameters. PETL will likely fail, too, if the full fine-tuning encounters the out-of-GPU-memory issue. This phenomenon happens because trainable parameters from these methods are generally entangled with the backbone, such that a lot of intermediate states have to be stored in GPU memory for gradient propagation. To alleviate this problem, we introduce Disentangled Transfer Learning (DTL), which disentangles the trainable parameters from the backbone using a lightweight Compact Side Network (CSN). By progressively extracting task-specific information with a few low-rank linear mappings and appropriately adding the information back to the backbone, CSN effectively realizes knowledge transfer in various downstream tasks. We conducted extensive experiments to validate the effectiveness of our method. The proposed method not only reduces a large amount of GPU memory usage and trainable parameters, but also outperforms existing PETL methods by a significant margin in accuracy, achieving new state-of-the-art on several standard benchmarks. The code is available at https://github.com/heekhero/DTL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑