注意源无关跨域少样本学习中的判别性陷阱
Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning
AI总结:
本文研究源无关跨域少样本学习中增强视觉模态判别性会抑制VLM性能的现象,提出通过扰动视觉学习以引导跨模态对齐的方法,实验显示在多种设置和数据集上取得新状态-of-the-art结果。
摘要:
源无关跨域少样本学习(SF-CDFSL)聚焦于在目标域有限训练数据上微调,如医学或卫星图像,其中视觉-语言模型(VLMs)如CLIP和SigLIP已表现出有希望的结果。传统视觉模型研究表明提高视觉判别性能提升性能。然而在VLM基于的SF-CDFSL任务中,我们发现增强视觉模态判别性实际上会抑制VLM性能。本文旨在深入探讨这一现象并提供解决方案。通过理论和实验证明,我们的研究揭示了使用典型交叉熵损失(L_vlm)微调本质上包含视觉学习部分和跨模态学习部分,其中跨模态部分对纠正SF-CDFSL中严重的模态错位至关重要。然而我们发现视觉学习本质上起着捷径作用,促使模型减少L_vlm而不考虑跨模态部分,从而阻碍跨模态对齐并损害性能。基于此解释,我们进一步提出解决该问题的方法:首先,我们扰动视觉学习以引导模型关注跨模态对齐。然后,我们利用视觉-文本语义关系在微调过程中逐步对齐视觉和文本模态。在各种设置、主干网络(CLIP、SigLip、PE-Core)和任务(4个CDFSL数据集和11个FSL数据集)上的广泛实验显示,我们一致设定了新的状态-of-the-art结果。代码可在https://github.com/zhenyuZ-HUST/CVPR26-Mind-the-Discriminability-Trap获取。
英文摘要:
Source-Free Cross-Domain Few-Shot Learning (SF-CDFSL) focuses on fine-tuning with limited training data from target domains (e.g., medical or satellite images), where Vision-Language Models (VLMs) such as CLIP and SigLIP have shown promising results. Current works in traditional visual models suggest that improving visual discriminability enhances performance. However, in VLM-based SF-CDFSL tasks, we find that \textbf{strengthening visual-modal discriminability actually suppresses VLMs' performance}. In this paper, we aim to delve into this phenomenon for an interpretation and a solution. By both theoretical and experimental proofs, our study reveals that fine-tuning with the typical cross-entropy loss ($\mathcal{L}_{\mathrm{vlm}}$) inherently includes a visual learning part and a cross-modal learning part, where the cross-modal part is crucial for rectifying the heavily disrupted modality misalignment in SF-CDFSL. However, we find that the visual learning essentially acts as a shortcut that encourages the model to reduce $\mathcal{L}_{\mathrm{vlm}}$ without considering the cross-modal part, therefore hindering the cross-modal alignment and harming the performance. Based on this interpretation, we further propose an approach to address this problem: first, we perturb the visual learning to guide the model to focus on the cross-modal alignment. Then, we use the visual-text semantic relationships to gradually align the visual and textual modalities during the fine-tuning. Extensive experiments on various settings, backbones (CLIP, SigLip, PE-Core), and tasks (4 CDFSL datasets and 11 FSL datasets) show that we consistently set new state-of-the-art results. Code is available at https://github.com/zhenyuZ-HUST/CVPR26-Mind-the-Discriminability-Trap.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。