Deep learning models are challenged by the distribution shift between the training data and test data. Recently, the large models pre-trained on diverse data have demonstrated unprecedented robustness to various distribution shifts. However, fine-tuning these models can lead to a trade-off between in-distribution (ID) performance and out-of-distribution (OOD) robustness. Existing methods for tackling this trade-off do not explicitly address the OOD robustness problem. In this paper, based on causal analysis of the aforementioned problems, we propose a novel fine-tuning method, which uses masked images as counterfactual samples that help improve the robustness of the fine-tuning model. Specifically, we mask either the semantics-related or semantics-unrelated patches of the images based on class activation map to break the spurious correlation, and refill the masked patches with patches from other images. The resulting counterfactual samples are used in feature-based distillation with the pre-trained model. Extensive experiments verify that regularizing the fine-tuning with the proposed masked images can achieve a better trade-off between ID and OOD performance, surpassing previous methods on the OOD performance. Our code is available at https://github.com/Coxy7/robust-finetuning.
Walking the Tightrope: Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuning
Australian Artificial Intelligence Institute (AAII)（澳大利亚人工智能研究所）；Faulty of Engineering and Information Technology（工程与信息技术学院）；University of Technology Sydney（悉尼技术大学）
Fine-tuning a Multiple Instance Learning Feature Extractor with Masked Context Modelling and Knowledge Distillation
Institute for Biomedical Informatics, Faculty of Medicine（医学院生物医学信息学研究所）；University Hospital Cologne, University of Cologne（科隆大学医院（科隆大学））；Center for Molecular Medicine Cologne (CMMC), Faculty of Medicine（科隆分子医学中心（CMMC，医学院））
Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget
Faculty of Computer Science, University of Vienna, Vienna, Austria（维也纳大学计算机科学学院）；UniVie Doctoral School Computer Science, University of Vienna（维也纳大学计算机科学博士学院）；Institute of Advanced Research in Artificial Intelligence (IARAI)（高级人工智能研究所）
提出 MAE-CT 方法，结合 MIM 的效率与 ID 的对象聚焦能力，通过 NNCLR 目标在预训练 MAE 顶层诱导语义聚类，无需标签且计算开销低，在 ViT 模型上多项指标达 SOTA。
MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning
Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）；Shanghai Jiao Tong University（上海交通大学）；West China Hospital, Sichuan University（四川大学华西医院）；Shanghai AI Laboratory（上海人工智能实验室）
Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；College of Information Technology, United Arab Emirates University（阿联酋大学信息技术学院）
MV-JAR: Masked Voxel Jigsaw and Reconstruction for LiDAR-Based Self-Supervised Pre-Training
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Carnegie Mellon University（卡内基梅隆大学）
本文介绍了基于LiDAR的自监督预训练的掩蔽体素拼图和重建（MV-Jigsaw and Reconstruction）方法，以及在Waymo数据集上精心设计的数据高效的3D目标检测基准。受下游3D对象检测器中的场景-体素-点层次结构的启发，我们设计了考虑场景中的体素分布和体素内的局部点分布的掩蔽和重建策略。我们采用了一种反向Fursed-Voxel-Sampling策略来解决LiDAR点的不均匀分布，并提出了MV-Voxel，它结合了两种技术来建模上述分布，从而获得了优异的性能。我们的实验揭示了以前数据高效实验的局限性，这些实验均匀地对每个LiDAR序列中不同数据比例的微调分裂进行采样，导致分裂之间的数据多样性相似。为了解决这个问题，我们提出了一个新的基准，该基准对场景序列进行采样以进行各种微调分割，确保足够的模型收敛并提供对预训练方法的更准确评估。在Waymo基准测试和KITTI数据集上的实验表明，MV-SVM在各种数据尺度上都能持续显著地提高3D检测性能，与从头开始训练相比，mAPH提高了6.3%。代码和基准将在https://github.com/SmartBot-PJLab/MV-JAR上提供。
Reinforcing Pre-trained Models Using Counterfactual Images
Hokkaido University（北海道大学）；Graduate School of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术研究院）；Faculty of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术学院）
DMC$^3$: Dual-Modal Counterfactual Contrastive Construction for Egocentric Video Question Answering
Nanjing University of Posts and Telecommunications（南京邮电大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Peng Cheng Laboratory（鹏城实验室）；MAIS, Institute of Automation, Chinese Academy of Sciences（自动化所MAIS）
CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale
The Key Laboratory of Smart Human-Computer Interaction（智能人机交互关键实验室）；Wearable Technology of Shaanxi Province, Xidian University, Xi'an, China（陕西 wearable 技术, 西安电子科技大学, 中国）；Nanjing University of Science and Technology（南京理工大学）
