思考还是不思考？面向视觉语言模型的强化学习选择性推理方法
Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models
AI总结：
针对视觉语言模型推理成本高的问题，提出两阶段TON训练策略，通过思维丢弃冷启动和GRPO优化实现选择性推理，可大幅降低token消耗且不损失甚至提升性能，为类人推理模式提供新思路。
摘要：
强化学习（RL）已被证明是提升视觉语言模型（VLM）推理能力的有效后训练策略。组相对策略优化（GRPO）是近期一种主流方法，它鼓励模型在作答前生成完整的推理轨迹，这会导致token使用量和计算成本上升。受人类类人思维过程启发——人们面对简单问题时会跳过推理，而在需要时会仔细思考——我们探索如何让VLM首先判断何时需要进行推理。为实现这一目标，我们提出TON，一种两阶段训练策略：（i）监督微调（SFT）阶段，采用简单且有效的“思维丢弃”操作，即随机将推理轨迹替换为空思维。该阶段引入了“思考与否”的格式，为选择性推理提供冷启动；（ii）GRPO阶段，让模型自由探索何时需要思考，同时最大化任务相关的结果奖励。实验结果表明，与原始GRPO相比，TON最多可将生成完成长度降低90%，且不会牺牲性能甚至能提升性能。在大语言模型任务（GSM8K）、视觉语言模型任务（CLEVR、Super-CLEVR、GeoQA）以及智能体任务（AITZ）上的进一步评估——涵盖3B和7B两种模型规模下的不同推理难度——一致显示，随着训练推进，模型会逐步学会跳过不必要的推理步骤。这些发现为在强化学习方法中实现类人推理模式指明了方向。我们的代码可在https://github.com/kokolerk/TON获取。
英文摘要：
Reinforcement Learning (RL) has proven to be an effective post-training strategy for enhancing reasoning in vision-language models (VLMs). Group Relative Policy Optimization (GRPO) is a recent prominent method that encourages models to generate complete reasoning traces before answering, leading to increased token usage and computational cost. Inspired by the human-like thinking process-where people skip reasoning for easy questions but think carefully when needed-we explore how to enable VLMs to first decide when reasoning is necessary. To realize this, we propose TON, a two-stage training strategy: (i) a supervised fine-tuning (SFT) stage with a simple yet effective 'thought dropout' operation, where reasoning traces are randomly replaced with empty thoughts. This introduces a think-or-not format that serves as a cold start for selective reasoning; (ii) a GRPO stage that enables the model to freely explore when to think or not, while maximizing task-aware outcome rewards. Experimental results show that TON can reduce the completion length by up to 90% compared to vanilla GRPO, without sacrificing performance or even improving it. Further evaluations across LLM (GSM8K), VLM (CLEVR, Super-CLEVR, GeoQA), and Agentic (AITZ) tasks-covering a range of reasoning difficulties under both 3B and 7B models-consistently reveal that the model progressively learns to bypass unnecessary reasoning steps as training advances. These findings shed light on the path toward human-like reasoning patterns in RL approaches. Our code is available at https://github.com/kokolerk/TON.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
