不完美的视觉编码器：视觉语言模型的高效鲁棒微调
Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models
AI总结：
针对开源VLMs依赖冻结CLIP编码器导致图像理解错误传播的问题，提出一种选择性局部更新视觉编码器的高效鲁棒微调方法，在修正错误的同时保持整体鲁棒性并支持持续少样本更新。
摘要：
视觉语言模型（VLMs）在视觉问答和图像描述生成方面展现出令人印象深刻的能力，充当视觉模型与语言模型之间的关键纽带。然而，现有的开源VLMs严重依赖预训练且冻结的视觉编码器（如CLIP）。尽管CLIP在多个领域具备鲁棒性，但仍存在不可忽视的图像理解错误。这些错误会传播至VLMs的响应中，导致次优性能。在我们的工作中，我们提出了一种高效且鲁棒的方法来更新VLMs内部的视觉编码器。该方法选择性且局部地更新编码器，在先前出现错误的数据上实现了显著的性能提升，同时保持了整体的鲁棒性。此外，我们展示了该方法在持续少样本更新中的有效性。理论支撑、通用性与计算效率是我们方法的特征。
英文摘要：
Vision language models (VLMs) demonstrate impressive capabilities in visual question answering and image captioning, acting as a crucial link between visual and language models. However, existing open-source VLMs heavily rely on pretrained and frozen vision encoders (such as CLIP). Despite CLIP's robustness across diverse domains, it still exhibits non-negligible image understanding errors. These errors propagate to the VLM responses, resulting in sub-optimal performance. In our work, we propose an efficient and robust method for updating vision encoders within VLMs. Our approach selectively and locally updates encoders, leading to substantial performance improvements on data where previous mistakes occurred, while maintaining overall robustness. Furthermore, we demonstrate the effectiveness of our method during continual few-shot updates. Theoretical grounding, generality, and computational efficiency characterize our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
