Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation (LoRA) can effectively adapt large pre-trained foundation models to downstream tasks using only a small fraction (0.1%-10%) of the original trainable weights. An under-explored question of PEFT is in extending the pre-training phase without supervised labels; that is, can we adapt a pre-trained foundation model to a new domain via efficient self-supervised pre-training on this domain? In this work, we introduce ExPLoRA, a highly effective technique to improve transfer learning of pre-trained vision transformers (ViTs) under domain shifts. Initializing a ViT with pre-trained weights on large, natural-image datasets such as from DinoV2 or MAE, ExPLoRA continues the unsupervised pre-training objective on a new domain, unfreezing 1-2 pre-trained ViT blocks and tuning all other layers with LoRA. We then fine-tune the resulting model only with LoRA on this new domain for supervised learning. Our experiments demonstrate state-of-the-art results on satellite imagery, even outperforming fully pre-training and fine-tuning ViTs. Using the DinoV2 training objective, we demonstrate up to 8% improvement in linear probing top-1 accuracy on downstream tasks while using <10% of the number of parameters that are used in prior fully-tuned state-of-the-art approaches. Our ablation studies confirm the efficacy of our approach over other baselines such as PEFT. Code is available on the project website: https://samar-khanna.github.io/ExPLoRA/
TORE: Token Recycling in Vision Transformers for Efficient Active Visual Exploration
University of Warsaw(华沙大学);Faculty of Mathematics and Computer Science, Jagiellonian University(雅盖隆大学数学与计算机科学学院);Ardigen SA(Ardigen SA公司);Center for Molecular Medicine Cologne (CMMC), University of Cologne(科隆大学科隆分子医学中心);IDEAS NCBR
An Exploratory Approach Towards Investigating and Explaining Vision Transformer and Transfer Learning for Brain Disease Detection
Ahsanullah University of Science and Technology(阿沙努拉科学与技术大学);Bangladesh University of Engineering and Technology(孟加拉工程与技术大学);Southeast University(东南大学)
Adaptive Token Sampling For Efficient Vision Transformers
Technische Universität Berlin(柏林工业大学);Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所);University of Bonn(波恩大学);Microsoft(微软);University of California, Davis(加州大学戴维斯分校);Meta Reality Labs(Meta现实实验室)
End-to-End Object Detection with Adaptive Clustering Transformer
Peking University(北京大学);Shanghai AI Laboratory(上海人工智能实验室);CUHK-SenseTime Joint Laboratory(香港中文大学-商汤科技联合实验室);The Chinese University of Hong Kong(香港中文大学)
Image Recognition with Online Lightweight Vision Transformer: A Survey
Beijing University of Posts and Telecommunications(北京邮电大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology, Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science(计算能力网络与信息安全重点实验室,教育部,山东计算机科学中心,齐鲁大学,山东省计算能力互联网与服务计算重点实验室,山东省计算机科学基础研究中心)
Safe Self-Refinement for Transformer-based Domain Adaptation
Stony Brook University(石溪大学);XPeng Motors(小鹏汽车)
无监督的领域自适应(UDA)旨在利用标签丰富的源域来解决相关未标记目标域上的任务。这是一个具有挑战性的问题,特别是当一个大的域之间的差距位于源和目标域。本文提出了一种新的解决方案SSRT(Safe Self-Refinement for Transformer-based领域自适应),从两个方面进行了改进。首先,由于视觉Transformer在各种视觉任务中的成功,我们为SSRT配备了一个Transformer骨干。我们发现,视觉Transformer与简单的对抗性适应的组合超过了最好的报告卷积神经网络(CNN)的结果,基于具有挑战性的DomainNet基准,显示其强大的可转移特征表示。其次,为了降低模型崩溃的风险,提高知识转移的有效性与大差距的领域之间,我们提出了一个安全的自我完善策略。具体而言,SSRT利用扰动的目标域数据的预测来细化模型。由于视觉Transformer的模型容量很大,并且在这种具有挑战性的任务中的预测可能是有噪声的,因此设计了一种安全的训练机制来自适应地调整学习配置。在几个广泛测试的UDA基准上进行了广泛的评估,SSRT始终获得最佳性能,包括在Bull-Home上的85.43%,在VisDA-2017上的88.76%和在DomainNet上的45.2%。
Learn Faster and Remember More: Balancing Exploration and Exploitation for Continual Test-time Adaptation
School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(电子、电气与通信工程学院,中国科学院大学);School of Computer Science and Technology, University of Chinese Academy of Sciences(计算机科学与技术学院,中国科学院大学);Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences(智能信息处理重点实验室,计算技术研究所)
DAPoinTr: Domain Adaptive Point Transformer for Point Cloud Completion
Deakin University(迪肯大学);Shanghai Jiao Tong University(上海交通大学);East China Normal University(华东师范大学);Zhejiang University(浙江大学);The University of Western Australia(西澳大学)