arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2504.11468cs.CL
收藏

SFT还是RL?对训练R1类推理大型视觉语言模型的早期研究

SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models

University of California, Santa Cruz(加州大学圣克鲁兹分校);University of Texas at Dallas(德克萨斯大学达拉斯分校);The Pennsylvania State University(宾夕法尼亚州立大学);Amazon Research(亚马逊研究院)

Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, Cihang Xie

AI总结:

本研究重新审视SFT+RL训练LVLMs的主流范式,发现SFT会诱导“伪推理路径”破坏后续RL;引入多模态数据集VLAA-Thinking系统研究该效应,提出基于GRPO混合奖励模块的RL方法,所训练模型在4B规模LVLMs推理任务中取得top-1性能。

摘要:

本研究重新审视了训练大型视觉语言模型(LVLMs)的主流范式——先监督微调(SFT)再强化学习(RL),并揭示了一个关键发现:SFT会诱导从专家模型模仿而来的“伪推理路径”,从而显著破坏后续的RL。这些路径虽然可能与RL模型的原生推理路径相似,但往往包含冗长、犹豫、信息量较少的步骤以及错误推理。为系统研究这一效应,我们引入VLAA-Thinking,一个专为支持LVLMs推理而设计的新型多模态数据集。该数据集通过包含 captioning、推理蒸馏、答案重写与验证的六步流程构建,包含用于SFT的高质量、逐步视觉推理轨迹,以及来自同一数据源的更具挑战性的RL拆分。利用该数据集,我们开展了对比SFT、RL及其组合的广泛实验。结果显示,SFT虽能帮助模型学习推理格式,但常将对齐模型锁定在模仿性、僵化的推理模式中,阻碍进一步学习。相比之下,基于集成感知与认知信号的新型混合奖励模块的Group Relative Policy Optimization(GRPO),我们的RL方法培养出更真实、自适应的推理行为。值得注意的是,基于Qwen2.5VL 3B的模型VLAA-Thinker在Open LMM推理排行榜(https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard)上的4B规模LVLMs中取得top-1性能,比此前的最先进模型高出1.8%。我们希望我们的发现能为开发具备推理能力的LVLMs提供宝贵见解,并为该领域未来研究提供参考。

英文摘要:

This work revisits the dominant supervised fine-tuning (SFT) then reinforcement learning (RL) paradigm for training Large Vision-Language Models (LVLMs), and reveals a key finding: SFT can significantly undermine subsequent RL by inducing ``pseudo reasoning paths'' imitated from expert models. While these paths may resemble the native reasoning paths of RL models, they often involve prolonged, hesitant, less informative steps, and incorrect reasoning. To systematically study this effect, we introduce VLAA-Thinking, a new multimodal dataset designed to support reasoning in LVLMs. Constructed via a six-step pipeline involving captioning, reasoning distillation, answer rewrite and verification, VLAA-Thinking comprises high-quality, step-by-step visual reasoning traces for SFT, along with a more challenging RL split from the same data source. Using this dataset, we conduct extensive experiments comparing SFT, RL and their combinations. Results show that while SFT helps models learn reasoning formats, it often locks aligned models into imitative, rigid reasoning modes that impede further learning. In contrast, building on the Group Relative Policy Optimization (GRPO) with a novel mixed reward module integrating both perception and cognition signals, our RL approach fosters more genuine, adaptive reasoning behavior. Notably, our model VLAA-Thinker, based on Qwen2.5VL 3B, achieves top-1 performance on Open LMM Reasoning Leaderboard (https://huggingface.co/spaces/opencompass/Open_LMM_Reasoning_Leaderboard) among 4B scale LVLMs, surpassing the previous state-of-the-art by 1.8%. We hope our findings provide valuable insights in developing reasoning-capable LVLMs and can inform future research in this area.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑