校准多模态表征：无需标注的群体鲁棒性探索
Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
AI总结：
针对预训练视觉-语言模型微调存在的计算成本高、依赖伪特征等问题，本文提出无需群体标注的轻量级表征校准方法，通过生成校准集并对比学习校准表征，有效提升CLIP的群体鲁棒性与泛化能力。
摘要：
对CLIP这类预训练视觉-语言模型进行微调已在多种下游任务上取得成功，但该范式仍存在若干痛点：（1）直接微调整个预训练模型既耗时又计算成本高，且微调后的模型往往高度特化，限制了实际部署的实用性；（2）近期研究表明，预训练视觉-语言分类器可能过度依赖伪特征——即训练数据中与目标相关但与真实标注函数无关的模式；（3）现有缓解伪特征依赖的研究大多基于可识别这类特征的假设，无法为实际应用提供明确保障。作为一项先导研究，本文聚焦于在不使用任何群体标注的情况下，缓解CLIP对伪特征的依赖问题。为此，我们系统研究了CLIP和CLIP+ERM中伪相关的存在性。首先，我们借鉴近期关于深度特征重加权（DFR）的研究，验证了最后层重训练可大幅提升预训练CLIP的群体鲁棒性。基于此，我们提出一种用于CLIP微调的轻量级表征校准方法：先利用预训练CLIP生成校准集，再通过对比学习校准该集合内样本的表征，全程无需群体标签。在多个基准上开展的大量实验与深入可视化验证了所提方法的有效性，大幅降低了模型对伪特征的依赖，显著提升了模型泛化能力。
英文摘要：
Fine-tuning pre-trained vision-language models, like CLIP, has yielded success on diverse downstream tasks. However, several pain points persist for this paradigm: (i) directly tuning entire pre-trained models becomes both time-intensive and computationally costly. Additionally, these tuned models tend to become highly specialized, limiting their practicality for real-world deployment; (ii) recent studies indicate that pre-trained vision-language classifiers may overly depend on spurious features -- patterns that correlate with the target in training data, but are not related to the true labeling function; and (iii) existing studies on mitigating the reliance on spurious features, largely based on the assumption that we can identify such features, does not provide definitive assurance for real-world applications. As a piloting study, this work focuses on exploring mitigating the reliance on spurious features for CLIP without using any group annotation. To this end, we systematically study the existence of spurious correlation on CLIP and CLIP+ERM. We first, following recent work on Deep Feature Reweighting (DFR), verify that last-layer retraining can greatly improve group robustness on pretrained CLIP. In view of them, we advocate a lightweight representation calibration method for fine-tuning CLIP, by first generating a calibration set using the pretrained CLIP, and then calibrating representations of samples within this set through contrastive learning, all without the need for group labels. Extensive experiments and in-depth visualizations on several benchmarks validate the effectiveness of our proposals, largely reducing reliance and significantly boosting the model generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
