arXiv 2505.146772025-10-26cs.CV

Visionary-R1：通过强化学习缓解视觉推理中的捷径学习

Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning

Hong Kong Baptist University（香港 Baptist 大学）；Shanghai AI Lab（上海人工智能实验室）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）

Jiaer Xia, Yuhang Zang, Peng Gao, Sharon Li, Kaiyang Zhou

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Visionary-R1，通过强化学习训练视觉语言模型，采用描述-推理-回答格式以缓解捷径学习，在273K无CoT问答对上训练后，在多个视觉推理基准上超越GPT-4o等强模型。

摘要：

学习通用推理能力一直是人工智能领域的一个长期挑战。近期对大型语言模型（LLM）的研究，如DeepSeek-R1，表明诸如GRPO之类的强化学习技术可以使预训练的LLM仅使用简单的问答对就能发展出推理能力。在本文中，我们旨在通过强化学习和视觉问答对来训练视觉语言模型（VLM）对图像数据进行推理，而无需任何显式的思维链（CoT）监督。我们的研究结果表明，简单地将强化学习应用于VLM——通过提示模型在给出答案之前生成推理链——可能会导致模型从简单问题中发展出捷径，从而降低其在未见数据分布上的泛化能力。我们认为，缓解捷径学习的关键在于鼓励模型在推理之前先解释图像。因此，我们训练模型遵循“描述-推理-回答”的输出格式：首先为图像生成详细的描述，然后构建详尽的推理链。当在273K个无CoT的视觉问答对上训练并仅使用强化学习时，我们的模型（名为Visionary-R1）在多个视觉推理基准上优于强大的多模态模型，如GPT-4o、Claude3.5-Sonnet和Gemini-1.5-Pro。

英文摘要：

Learning general-purpose reasoning capabilities has long been a challenging problem in AI. Recent research in large language models (LLMs), such as DeepSeek-R1, has shown that reinforcement learning techniques like GRPO can enable pre-trained LLMs to develop reasoning capabilities using simple question-answer pairs. In this paper, we aim to train visual language models (VLMs) to perform reasoning on image data through reinforcement learning and visual question-answer pairs, without any explicit chain-of-thought (CoT) supervision. Our findings indicate that simply applying reinforcement learning to a VLM -- by prompting the model to produce a reasoning chain before providing an answer -- can lead the model to develop shortcuts from easy questions, thereby reducing its ability to generalize across unseen data distributions. We argue that the key to mitigating shortcut learning is to encourage the model to interpret images prior to reasoning. Therefore, we train the model to adhere to a caption-reason-answer output format: initially generating a detailed caption for an image, followed by constructing an extensive reasoning chain. When trained on 273K CoT-free visual question-answer pairs and using only reinforcement learning, our model, named Visionary-R1, outperforms strong multimodal models, such as GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro, on multiple visual reasoning benchmarks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。