通过对比的视角：VLMs中的自改进视觉推理
Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs
AI总结：
通过视觉对比提升VLMs的推理能力，提出VC-STaR框架，有效减少推理幻觉并提升多种VLMs的视觉推理性能。
摘要：
推理能力已成为大语言模型的关键能力。在语言任务中，这种能力可以通过自我改进技术来增强，该技术通过细化推理路径来为后续微调做准备。然而，将这些语言基础的自我改进方法扩展到视觉语言模型（VLMs）则面临独特挑战：推理路径中的视觉幻觉无法有效验证或修正。我们的解决方案基于一个关于视觉对比的关键观察：当给出一个对比性VQA对，即两个视觉相似的图像和同义问题时，VLMs能够更精确地识别相关视觉线索。受这一观察启发，我们提出了视觉对比自教推理器（VC-STaR），一种新的自我改进框架，利用视觉对比来减轻模型生成的推理中的幻觉。我们收集了多样化的VQA数据集，根据多模态相似性筛选对比对，并使用VC-STaR生成推理。因此，我们获得了一个新的视觉推理数据集VisCoR-55K，然后通过监督微调提升各种VLMs的推理能力。广泛的实验表明，VC-STaR不仅优于现有的自我改进方法，还超过了在SoTA视觉推理数据集上微调的模型，证明了VLMs固有的对比能力可以自我提升其视觉推理能力。项目地址：https://github.com/zhiyupan42/VC-STaR。
英文摘要：
Reasoning has emerged as a key capability of large language models. In linguistic tasks, this capability can be enhanced by self-improving techniques that refine reasoning paths for subsequent finetuning. However, extending these language-based self-improving approaches to vision language models (VLMs) presents a unique challenge:~visual hallucinations in reasoning paths cannot be effectively verified or rectified. Our solution starts with a key observation about visual contrast: when presented with a contrastive VQA pair, i.e., two visually similar images with synonymous questions, VLMs identify relevant visual cues more precisely. Motivated by this observation, we propose Visual Contrastive Self-Taught Reasoner (VC-STaR), a novel self-improving framework that leverages visual contrast to mitigate hallucinations in model-generated rationales. We collect a diverse suite of VQA datasets, curate contrastive pairs according to multi-modal similarity, and generate rationales using VC-STaR. Consequently, we obtain a new visual reasoning dataset, VisCoR-55K, which is then used to boost the reasoning capability of various VLMs through supervised finetuning. Extensive experiments show that VC-STaR not only outperforms existing self-improving approaches but also surpasses models finetuned on the SoTA visual reasoning datasets, demonstrating that the inherent contrastive ability of VLMs can bootstrap their own visual reasoning. Project at: https://github.com/zhiyupan42/VC-STaR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
