保留预训练VLM的多模态能力以提升视觉-语言组合性
Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality
AI总结：
针对传统微调提升VLM组合推理会损害多模态能力的问题，提出融合局部硬负样本损失与选择性校准正则化的FSC-CLIP，在保持多模态能力的同时实现了与SOTA相当的组合性。
摘要：
本文提出一种新方法，可在不牺牲零样本多模态任务性能的前提下，增强预训练视觉语言模型（VLM）的组合理解能力。传统微调方法通常以降低多模态能力为代价提升组合推理能力，主要原因是使用了全局硬负样本（HN）损失，该损失会对比图像和文本的全局表示。这种全局HN损失会推动与原文本高度相似的硬负样本文本，破坏模型的多模态表示。为克服这一局限，我们提出细粒度选择性校准CLIP（FSC-CLIP），该方法融合了局部硬负样本损失和选择性校准正则化。这些创新在保留模型表示完整性的同时，提供了细粒度的负样本监督。我们在组合性和多模态任务的多种基准上开展了广泛评估，结果表明FSC-CLIP不仅能达到与当前最优模型相当的组合性，还能保留强大的多模态能力。代码可在以下链接获取：https://github.com/ytaek-oh/fsc-clip。
英文摘要：
In this paper, we propose a new method to enhance compositional understanding in pre-trained vision and language models (VLMs) without sacrificing performance in zero-shot multi-modal tasks. Traditional fine-tuning approaches often improve compositional reasoning at the cost of degrading multi-modal capabilities, primarily due to the use of global hard negative (HN) loss, which contrasts global representations of images and texts. This global HN loss pushes HN texts that are highly similar to the original ones, damaging the model's multi-modal representations. To overcome this limitation, we propose Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard negative loss and selective calibrated regularization. These innovations provide fine-grained negative supervision while preserving the model's representational integrity. Our extensive evaluations across diverse benchmarks for both compositionality and multi-modal tasks show that FSC-CLIP not only achieves compositionality on par with state-of-the-art models but also retains strong multi-modal capabilities. Code is available at: https://github.com/ytaek-oh/fsc-clip.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
