CLIP-DPO：视觉语言模型作为修复LVLM幻觉的偏好来源
CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs
AI总结：
提出CLIP-DPO方法，利用CLIP的偏好信号优化LVLM，无需额外数据或API，显著减少幻觉并保持基准性能。
摘要：
尽管近期取得了成功，但LVLM（大型视觉语言模型）容易产生幻觉，例如虚构物体及其属性或关系等细节，这限制了其在实际应用中的部署。为了解决这一问题并提高其鲁棒性，我们提出了CLIP-DPO，一种偏好优化方法，利用对比预训练的视觉语言（VL）嵌入模型（如CLIP）进行基于DPO的LVLM优化。与先前解决LVLM幻觉的工作不同，我们的方法不依赖付费API，也不需要额外的训练数据或部署其他外部LVLM。相反，从初始的监督微调数据池出发，我们生成多样化的预测，并根据其CLIP图像-文本相似度进行排序，然后使用基于规则的稳健过滤方法，获得用于DPO训练的正负样本对。我们将CLIP-DPO微调应用于MobileVLM-v2系列模型和LlaVA-1.5，在所有情况下，与基线模型相比，幻觉减少方面均观察到显著改进。我们还观察到零样本分类性能更好，表明接地能力有所提升，并验证了在标准LVLM基准上的原始性能总体得以保持。
英文摘要：
Despite recent successes, LVLMs or Large Vision Language Models are prone to hallucinating details like objects and their properties or relations, limiting their real-world deployment. To address this and improve their robustness, we present CLIP-DPO, a preference optimization method that leverages contrastively pre-trained Vision-Language (VL) embedding models, such as CLIP, for DPO-based optimization of LVLMs. Unlike prior works tackling LVLM hallucinations, our method does not rely on paid-for APIs, and does not require additional training data or the deployment of other external LVLMs. Instead, starting from the initial pool of supervised fine-tuning data, we generate a diverse set of predictions, which are ranked based on their CLIP image-text similarities, and then filtered using a robust rule-based approach to obtain a set of positive and negative pairs for DPO-based training. We applied CLIP-DPO fine-tuning to the MobileVLM-v2 family of models and to LlaVA-1.5, in all cases observing significant improvements in terms of hallucination reduction over baseline models. We also observe better performance for zero-shot classification, suggesting improved grounding capabilities, and verify that the original performance on standard LVLM benchmarks is overall preserved.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
