arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.20752cs.CVcs.AI
收藏

Reason-RFT:面向视觉语言模型视觉推理的强化微调

Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机科学学院,北京大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院自动化研究所人工智能学院)

Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Xiansheng Chen, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang

AI总结:

针对现有视觉语言模型思维链监督微调易过拟合、泛化性差的问题,提出两阶段强化微调框架Reason-RFT,结合监督微调和组相对策略优化强化学习,在多维度基准上实现了性能、泛化性和数据效率的提升。

摘要:

视觉推理能力在理解复杂多模态数据中发挥着关键作用,推动着特定领域应用和通用人工智能(AGI)的发展。现有方法通过使用精心标注数据的思维链(CoT)监督微调来增强视觉语言模型(VLM)。然而,这种方法可能导致过拟合和认知僵化,限制了模型在领域迁移下的泛化能力,降低了实际应用价值。为克服这些局限,我们提出Reason-RFT,一种面向视觉推理的两阶段强化微调框架。首先,使用经过筛选的CoT数据进行监督微调(SFT),激活VLM的推理潜力。随后采用基于组相对策略优化(GRPO)的强化学习,生成多组推理-响应对以增强对领域迁移的适应性。为评估Reason-RFT,我们重构了一个涵盖视觉计数、结构感知和空间变换的综合数据集,作为从三个关键维度进行系统评估的基准。实验结果凸显了三大优势:(1)性能提升,Reason-RFT取得了当前最优结果,表现优于开源和专有模型;(2)泛化优势,在各类任务的领域迁移下仍保持稳健性能;(3)数据高效,在少样本学习场景中表现出色,超越了全数据集SFT基线。Reason-RFT为视觉推理引入了一种全新的训练范式,标志着多模态研究的重要进展。项目网站:https://tanhuajie.github.io/ReasonRFT

英文摘要:

Visual reasoning abilities play a crucial role in understanding complex multimodal data, advancing both domain-specific applications and artificial general intelligence (AGI). Existing methods enhance Vision-Language Models (VLMs) through Chain-of-Thought (CoT) supervised fine-tuning using meticulously annotated data. However, this approach may lead to overfitting and cognitive rigidity, limiting the model's generalization ability under domain shifts and reducing real-world applicability. To overcome these limitations, we propose Reason-RFT, a two-stage reinforcement fine-tuning framework for visual reasoning. First, Supervised Fine-Tuning (SFT) with curated CoT data activates the reasoning potential of VLMs. This is followed by reinforcement learning based on Group Relative Policy Optimization (GRPO), which generates multiple reasoning-response pairs to enhance adaptability to domain shifts. To evaluate Reason-RFT, we reconstructed a comprehensive dataset covering visual counting, structural perception, and spatial transformation, serving as a benchmark for systematic assessment across three key dimensions. Experimental results highlight three advantages: (1) performance enhancement, with Reason-RFT achieving state-of-the-art results and outperforming both open-source and proprietary models; (2) generalization superiority, maintaining robust performance under domain shifts across various tasks; and (3) data efficiency, excelling in few-shot learning scenarios and surpassing full-dataset SFT baselines. Reason-RFT introduces a novel training paradigm for visual reasoning and marks a significant step forward in multimodal research. Project website: https://tanhuajie.github.io/ReasonRFT

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑