复兴领域内微调方法以适用于无源跨领域少样本学习
Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
AI总结：
本文研究跨领域少样本学习中领域内微调方法的性能，发现LoRA等适配器方法优于提示方法，提出Semantic Probe框架提升模态对齐和类别分离。
摘要：
跨领域少样本学习（CDFSL）旨在通过有限样本适应大规模预训练模型到专业目标领域，但视觉-语言模型如CLIP的少样本微调仍不充分。通过建立多个CLIP的微调基线，发现适配器方法（如LoRA）在CDFSL中优于提示方法（如MaPLe），与领域内场景相反。为使这些有效领域内方法在CDFSL中竞争，我们分析现象并发现LoRA的优越性源于修复视觉CLS标记的坍塌注意力，通过聚焦文本相关视觉区域增强模态对齐和类别分离。进一步发现文本EOS标记对视觉样本的注意力更强，而CLIP的标准对比损失对模态对齐约束较弱。基于此，我们提出Semantic Probe框架，为适配器和提示方法提供插件式注意力修复方案。在四个CDFSL基准上的实验验证了我们的推理，实现了最先进的性能，并受益于两种微调范式。代码将发布。
英文摘要：
Cross-Domain Few-Shot Learning (CDFSL) aims to adapt large-scale pretrained models to specialized target domains with limited samples, yet the few-shot fine-tuning of vision-language models like CLIP remains underexplored. By establishing multiple fine-tuning baselines of CLIP for CDFSL, we find adapter-based methods (e.g., LoRA) consistently outperform prompt-based ones (e.g., MaPLe), contrary to in-domain scenarios. To make those effective in-domain methods competitive again in CDFSL, we analyze this phenomenon and discover LoRA's superiority stems from rectifying the collapsed attention of visual CLS token, enhancing modality alignment and class separation by focusing on text-related visual regions. Further, we find textual EOS token exhibit much better attention to visual samples, and CLIP's standard contrastive loss weakly constrains modality alignment. Based on these insights, we propose Semantic Probe, a plug-and-play attention rectification framework for both adapter- and prompt-based methods. Extensive experiments on four CDFSL benchmarks validate our rationale, achieving state-of-the-art performance and benefiting both fine-tuning paradigms. Codes will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
