SAIL-RL: 通过双奖励强化学习调优引导MLLM在何时以及如何思考
SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuning
AI总结：
SAIL-RL通过双奖励强化学习调优，提升多模态大语言模型的推理能力和可靠性。
摘要：
我们引入了SAIL-RL，一种强化学习（RL）后训练框架，通过教授多模态大语言模型（MLLMs）何时以及如何思考来增强其推理能力。现有方法受限于仅以结果为导向的监督，这种监督仅奖励正确的答案而无法确保合理的推理，以及统一的思考策略，往往导致在简单任务上过度思考而在复杂任务上不足。SAIL-RL通过双奖励系统解决了这些挑战：思考奖励通过事实基础、逻辑连贯性和答案一致性来评估推理质量，判断奖励则自适应地确定是否需要深入推理或直接回答。在最先进的SAIL-VL2上进行的实验表明，SAIL-RL在4B和8B规模上提高了推理和多模态理解基准测试的表现，实现了与商业闭源模型如GPT-4o的竞争性能，并显著减少了幻觉，确立了其作为构建更可靠和适应性强的MLLMs的原理性框架。代码将在https://github.com/BytedanceDouyinContent/SAIL-RL上提供。
英文摘要：
We introduce SAIL-RL, a reinforcement learning (RL) post-training framework that enhances the reasoning capabilities of multimodal large language models (MLLMs) by teaching them when and how to think. Existing approaches are limited by outcome-only supervision, which rewards correct answers without ensuring sound reasoning, and by uniform thinking strategies, which often lead to overthinking on simple tasks and underthinking on complex ones. SAIL-RL addresses these challenges with a dual reward system: the Thinking Reward, which evaluates reasoning quality through factual grounding, logical coherence, and answer consistency, and the Judging Reward, which adaptively determines whether deep reasoning or direct answering is appropriate. Experiments on the state-of-the-art SAIL-VL2 show that SAIL-RL improves reasoning and multimodal understanding benchmarks at both 4B and 8B scales, achieving competitive performance against commercial closed-source models such as GPT-4o, and substantially reduces hallucinations, establishing it as a principled framework for building more reliable and adaptive MLLMs. The code will be available at https://github.com/BytedanceDouyinContent/SAIL-RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
