当预训练模型规模快速增大时,在下游任务上进行微调的成本也在稳步上升。为了经济地微调这些模型,研究者提出了参数高效迁移学习(PETL,parameter-efficient transfer learning)方法,这类方法仅调整极小一部分可训练参数,就能高效学习高质量表征。然而,当前的PETL方法面临着一个困境:训练过程中GPU显存占用并未随可训练参数减少而有效降低。如果全量微调会遇到GPU显存不足的问题,PETL同样可能失败。出现这一现象的原因是,这些方法的可训练参数通常与骨干网络耦合在一起,因此必须在GPU显存中存储大量中间状态以用于梯度传播。为缓解这一问题,我们提出了解耦迁移学习(DTL,Disentangled Transfer Learning),该方法利用轻量级的紧凑侧网络(CSN,Compact Side Network)将可训练参数与骨干网络解耦。通过少量低秩线性映射逐步提取任务特定信息,并将该信息适当地加回骨干网络,CSN在各类下游任务中有效实现了知识迁移。我们开展了大量实验以验证该方法的有效性。所提方法不仅大幅降低了GPU显存占用量和可训练参数量,还在准确率上显著优于现有PETL方法,在多个标准基准上取得了新的最优性能。代码可在https://github.com/heekhero/DTL获取。
英文摘要:
When pre-trained models become rapidly larger, the cost of fine-tuning on downstream tasks steadily increases, too. To economically fine-tune these models, parameter-efficient transfer learning (PETL) is proposed, which only tunes a tiny subset of trainable parameters to efficiently learn quality representations. However, current PETL methods are facing the dilemma that during training the GPU memory footprint is not effectively reduced as trainable parameters. PETL will likely fail, too, if the full fine-tuning encounters the out-of-GPU-memory issue. This phenomenon happens because trainable parameters from these methods are generally entangled with the backbone, such that a lot of intermediate states have to be stored in GPU memory for gradient propagation. To alleviate this problem, we introduce Disentangled Transfer Learning (DTL), which disentangles the trainable parameters from the backbone using a lightweight Compact Side Network (CSN). By progressively extracting task-specific information with a few low-rank linear mappings and appropriately adding the information back to the backbone, CSN effectively realizes knowledge transfer in various downstream tasks. We conducted extensive experiments to validate the effectiveness of our method. The proposed method not only reduces a large amount of GPU memory usage and trainable parameters, but also outperforms existing PETL methods by a significant margin in accuracy, achieving new state-of-the-art on several standard benchmarks. The code is available at https://github.com/heekhero/DTL.
Disentangled Variational Representation for Heterogeneous Face Recognition
Center for Research on Intelligent Perception and Computing (CRIPAC), CASIA(中国科学院自动化研究所智能感知与计算研究中心);National Laboratory of Pattern Recognition (NLPR), CASIA(中国科学院自动化研究所模式识别国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Johns Hopkins University(约翰斯·霍普金斯大学)
DrFER: Learning Disentangled Representations for 3D Facial Expression Recognition
School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Dyn-Adapter: Towards Disentangled Representation for Efficient Visual Recognition
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University(上海交通大学图像通信与网络工程研究所);Chinese Academy of Sciences(中国科学院);Meituan Inc.(美团公司);Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University(上海交通大学合作 medianet 创新中心)
Look Across Elapse: Disentangled Representation Learning and Photorealistic Cross-Age Face Synthesis for Age-Invariant Face Recognition
National University of Singapore(新加坡国立大学);National University of Defense Technology(中国人民解放军国防科技大学);Panasonic R&D Center Singapore(松下新加坡研发中心);Nanyang Technological University(南洋理工大学);Inception Institute of Artificial Intelligence(Inception人工智能研究所);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Qihoo 360 AI Institute(奇虎360人工智能研究院)
Disentangled Source-Free Personalization for Facial Expression Recognition with Neutral Target Data
LIVIA, Dept. of Systems Engineering, ETS Montreal, Canada(LIVIA,系统工程系,蒙特利尔大学工程学院,加拿大);LIVIA, Dept. of Software and IT Engineering, ETS Montreal, Canada(LIVIA,软件与信息技术工程系,蒙特利尔大学工程学院,加拿大);Dept. of Health, Kinesiology & Applied Physiology, Concordia University, Montreal, Canada(健康、运动科学与应用生理学系,康科迪亚大学,蒙特利尔,加拿大)
CDNet: Contrastive Disentangled Network for Fine-Grained Image Categorization of Ocular B-Scan Ultrasound
Hangzhou Dianzi University(杭州电子科技大学);the Second Affiliated Hospital of Zhejiang University(浙江大学医学院附属第二医院);Waseda University(早稻田大学);Communication University of Zhejiang(浙江传媒学院)
Heterogeneous Face Recognition via Face Synthesis with Identity-Attribute Disentanglement
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Disentangled Action Recognition with Knowledge Bases
University of California, Berkeley(加州大学伯克利分校);Amazon Alexa AI(亚马逊Alexa人工智能部门);Fujitsu Laboratories Ltd.(富士通实验室有限公司);Pennsylvania State University(宾夕法尼亚州立大学)
LIAAD: Lightweight Attentive Angular Distillation for Large-scale Age-Invariant Face Recognition
University of Arkansas(阿肯色大学);Concordia University(康考迪亚大学)
解纠缠表示已被普遍采用的非线性不变人脸识别(AiFR)任务。然而,这些方法已经达到了一些局限性:(1)需要具有年龄标签的大规模人脸识别(FR)训练数据,这在实践中是有限的;(2)高性能的重型深度网络架构;以及(3)它们的评估通常发生在年龄相关的人脸数据库上,而忽略了标准的大规模FR数据库以保证鲁棒性。这项工作提出了一种新的轻量级注意角蒸馏(LIAAD)方法来克服这些限制。给定两个高性能的重型网络作为具有不同专业知识的教师,LIAAD引入了一种学习范式,可以有效地将这些教师的年龄不变的专注和角度知识提取到轻量级的学生网络中,使其更强大,具有更高的FR准确性和鲁棒性。因此,LIAAD方法能够利用具有和不具有年龄标签的FR数据集来训练AiFR模型。与以往主要关注闭集问题的精度和压缩比的提取方法不同,我们的LIAAD旨在解决开集问题,即大规模人脸识别。在LFW、IJB-B和IJB-C Janus、MFDB和MegaFace-FGNet上使用100万个干扰项进行的实验表明,该方法对轻量结构的识别是有效的。这项工作还提出了一个新的纵向面部老化(LogiFace)数据库\footnote{This database will be made available},为进一步研究与年龄相关的面部问题在未来。
D$^2$ST-Adapter: Disentangled-and-Deformable Spatio-Temporal Adapter for Few-shot Action Recognition
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);Shenyang Institute of Automation, Chinese Academy of Sciences(沈阳自动化研究所,中国科学院)
Learning Fair Face Representation With Progressive Cross Transformer
Nanjing University of Science and Technology(南京理工大学);University of Chinese Academy of Sciences(中国科学院大学);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective
Jiangsu University(江苏大学);Griffith University(格里菲斯大学);National University of Singapore(新加坡国立大学);University of New South Wales(新南威尔士大学);Fujian University of Technology(福建理工大学);Xi’an Jiaotong-Liverpool University(西交利物浦大学);German Research Center for Artificial Intelligence(德国人工智能研究中心);Tsinghua University(清华大学);Technical University of Munich(慕尼黑工业大学);Imperial College London(伦敦帝国学院);Queen Mary University of London(伦敦玛丽女王大学)
Disentangling Semantic-to-visual Confusion for Zero-shot Learning
Suzhou University of Science and Technology(苏州科技大学);Tianjin University(天津大学);Suzhou Institute of Trade & Commerce(苏州经贸职业技术学院);Xi’an Jiaotong-Liverpool University(西交利物浦大学)
Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Tencent(腾讯);National University of Singapore(新加坡国立大学);Research Center of Artificial Intelligence, Peng Cheng Laboratory(鹏城实验室人工智能研究中心);Shenzhen Qiji Technology Co., Ltd.(深圳奇基科技有限公司)