Visual Parameter-Efficient Fine-Tuning (PEFT) has become a powerful alternative for full fine-tuning so as to adapt pre-trained vision models to downstream tasks, which only tunes a small number of parameters while freezing the vast majority ones to ease storage burden and optimization difficulty. However, existing PEFT methods introduce trainable parameters to the same positions across different tasks depending solely on human heuristics and neglect the domain gaps. To this end, we study where to introduce and how to allocate trainable parameters by proposing a novel Sensitivity-aware visual Parameter-efficient fine-Tuning (SPT) scheme, which adaptively allocates trainable parameters to task-specific important positions given a desired tunable parameter budget. Specifically, our SPT first quickly identifies the sensitive parameters that require tuning for a given task in a data-dependent way. Next, our SPT further boosts the representational capability for the weight matrices whose number of sensitive parameters exceeds a pre-defined threshold by utilizing existing structured tuning methods, e.g., LoRA [23] or Adapter [22], to replace directly tuning the selected sensitive parameters (unstructured tuning) under the budget. Extensive experiments on a wide range of downstream recognition tasks show that our SPT is complementary to the existing PEFT methods and largely boosts their performance, e.g., SPT improves Adapter with supervised pre-trained ViT-B/16 backbone by 4.2% and 1.4% mean Top-1 accuracy, reaching SOTA performance on FGVC and VTAB-1k benchmarks, respectively. Source code is at https://github.com/ziplab/SPT
Bosch Center for AI(博世人工智能中心);University of Tübingen(蒂宾根大学);University of Mannheim(曼海姆大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);Bosch Center for Artificial Intelligence(博世人工智能中心)
Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models
Harvard University(哈佛大学);Xi’an University of Science and Technology(西安科技大学);University of Chinese Academy of Sciences(中国科学院大学);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);University of Chicago(芝加哥大学);Yunnan University(云南大学);Tsinghua University(清华大学)
LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning
University of Electronic Science and Technology of China(电子科技大学);The University of Melbourne(墨尔本大学);Nankai University(南开大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
PAT: Pruning-Aware Tuning for Large Language Models
School of Electronic Science and Engineering, Nanjing University(南京大学电子科学与工程学院);University of Arizona(亚利桑那大学);Samsung Electronic Research Centre of China(三星中国电子研究中心);Interdisciplinary Research Center for Future Intelligent Chips, Nanjing University(南京大学未来智能芯片交叉研究中心)
Confounder-Aware Medical Data Selection for Fine-Tuning Pretrained Vision Models
West China Hospital, Sichuan University(四川大学华西医院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);College of Computer Science, Sichuan University(四川大学计算机学院);School of Biomedical Engineering, University of Science and Technology of China(中国科学技术大学生物医学工程学院)
University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Tulane University(路易斯安那州立大学);Carnegie Mellon University(卡内基梅隆大学);Oak Ridge National Laboratory(橡树岭国家实验室);Georgia Institute of Technology(佐治亚理工学院)
Alibaba Group(阿里巴巴集团);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Zhejiang University(浙江大学)
对比图像预训练(CLIP)模型具有zero-shot能力,通过使用图像与提示语句“a [CONTEXT] of [CLASS]"之间的相似性来对属于“[CLASS]”的图像进行分类。基于“[CONTEXT]"中的详尽文本线索,CLIP模型能够感知不同的上下文,例如背景、风格、视点,并对大范围的分布偏移表现出前所未有的鲁棒性。然而,最近的工作发现CLIP模型的进一步微调提高了准确性,但牺牲了下游任务的鲁棒性。我们进行了一项实证调查,以表明微调会破坏预训练CLIP特征的上下文感知能力。为了解决这个问题,我们提出了上下文感知鲁棒微调(CAR-FT)。CAR-FT在微调期间对模型进行规则化,以捕获上下文信息。具体来说,我们使用zero-shot提示权重来获得图像中包含的上下文分布。CAR-FT通过最小化由原始/微调CLIP模型引起的上下文分布之间的Kullback-Leibler分歧(KLD),使CLIP的上下文感知能力继承到下游任务中,并实现更高的分布内(ID)和分布外(OOD)准确性。实验结果表明,CAR-FT在ImageNet的五个OOD测试数据集上具有出色的鲁棒性,同时在九个下游任务上带来了准确率的提高。此外,CAR-FT超越了以前的领域泛化(DG)方法,在DomainBed基准测试中获得了78.5%的平均准确率,建立了新的最先进的水平。
TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model
Institute of Automation, CAS(中国科学院自动化研究所);State Key Laboratory of Multimodal Artificial Intelligence Systems(多模态人工智能系统国家重点实验室);State Key Lab of Processors(处理器国家重点实验室);Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detection
QUT Centre for Robotics, Queensland University of Technology(昆士兰科技大学QUT机器人中心);Australian Institute for Machine Learning, University of Adelaide(阿德莱德大学澳大利亚机器学习研究所)
SAFT: Sensitivity-Aware Filtering and Transmission for Adaptive 3D Point Cloud Communication over Wireless Channels
School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院);Key Laboratory of Machine Intelligence and System Control, Ministry of Education(教育部机器智能与系统控制重点实验室);School of Computer Science and Technology, Shandong University(山东大学计算机科学与技术学院)
Sensitivity-Aware Post-Training Quantization for Deep Neural Networks
South China University of Technology(华南理工大学);Pazhou Laboratory(Pazhou实验室);Key Laboratory of Big Data and Intelligent Robot, Ministry of Education(大数据与智能机器人重点实验室,教育部)
SILT: Shadow-aware Iterative Label Tuning for Learning to Detect Shadows from Noisy Labels
The Chinese University of Hong Kong(香港中文大学);The Shun Hing Institute of Advanced Engineering(信兴高等工程研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)