arXiv 2411.182032025-04-23cs.CVcs.CL

Critic-V：VLM评论家帮助捕捉多模态推理中的VLM错误

Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning

Fudan University（复旦大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiaotong University（上海交通大学）；Nankai University（南开大学）；Shanghai University（上海大学）；Nanyang Technological University（南洋理工大学）；Hong Kong Polytechnic University（香港理工大学）；Tongji University（同济大学）；Nanjing University（南京大学）；University of California, Merced（加州大学梅德福分校）；Chinese University of Hong Kong（香港中文大学）

Di Zhang, Junxian Li, Jingdi Lei, Xunzhi Wang, Yujie Liu, Zonglin Yang, Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, Peng Ye, Wanli Ouyang, Dongzhan Zhou

备注：: 16 pages, 11 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

Critic-V提出一种受Actor-Critic启发的框架，通过Reasoner生成推理路径和Critic提供自然语言批评来增强视觉语言模型的多模态推理能力，在8个基准中5个上超越GPT-4V等现有方法。

摘要：

视觉语言模型（VLM）在多模态推理任务中展现出显著的进步。然而，由于幻觉图像理解或未优化的推理路径等问题，它们仍然经常生成不准确或不相关的响应。为了解决这些挑战，我们引入了Critic-V，一个受Actor-Critic范式启发的新框架，以增强VLM的推理能力。该框架通过整合两个独立组件来解耦推理过程和评论过程：Reasoner，基于视觉和文本输入生成推理路径；以及Critic，提供建设性批评以优化这些路径。在这种方法中，Reasoner根据文本提示生成推理响应，这些响应可以基于Critic的反馈作为策略迭代演化。这一交互过程在理论上由强化学习框架驱动，其中Critic提供自然语言批评而非标量奖励，从而能够提供更细致的反馈以增强Reasoner在复杂推理任务上的能力。Critic模型使用直接偏好优化（DPO）进行训练，利用由基于规则的奖励（RBR）排序的批评偏好数据集来增强其评论能力。评估结果表明，Critic-V框架在8个基准中的5个上显著优于现有方法，包括GPT-4V，特别是在推理准确性和效率方面。将Reasoner的动态基于文本策略与偏好优化的Critic的建设性反馈相结合，实现了更可靠且上下文敏感的多模态推理过程。我们的方法为增强VLM的可靠性提供了有前景的解决方案，提高了它们在诸如自动驾驶和具身智能等真实世界推理密集型多模态应用中的性能。

英文摘要：

Vision-language models (VLMs) have shown remarkable advancements in multimodal reasoning tasks. However, they still often generate inaccurate or irrelevant responses due to issues like hallucinated image understandings or unrefined reasoning paths. To address these challenges, we introduce Critic-V, a novel framework inspired by the Actor-Critic paradigm to boost the reasoning capability of VLMs. This framework decouples the reasoning process and critic process by integrating two independent components: the Reasoner, which generates reasoning paths based on visual and textual inputs, and the Critic, which provides constructive critique to refine these paths. In this approach, the Reasoner generates reasoning responses according to text prompts, which can evolve iteratively as a policy based on feedback from the Critic. This interaction process was theoretically driven by a reinforcement learning framework where the Critic offers natural language critiques instead of scalar rewards, enabling more nuanced feedback to boost the Reasoner's capability on complex reasoning tasks. The Critic model is trained using Direct Preference Optimization (DPO), leveraging a preference dataset of critiques ranked by Rule-based Reward~(RBR) to enhance its critic capabilities. Evaluation results show that the Critic-V framework significantly outperforms existing methods, including GPT-4V, on 5 out of 8 benchmarks, especially regarding reasoning accuracy and efficiency. Combining a dynamic text-based policy for the Reasoner and constructive feedback from the preference-optimized Critic enables a more reliable and context-sensitive multimodal reasoning process. Our approach provides a promising solution to enhance the reliability of VLMs, improving their performance in real-world reasoning-heavy multimodal applications such as autonomous driving and embodied intelligence.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。