arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2504.16656cs.CV
收藏

Skywork R1V2:面向推理的多模态混合强化学习

Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoning

Skywork AI;Kunlun Inc.(Kunlun公司)

Peiyu Wang, Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, Yang Liu, Yahui Zhou

AI总结:

本文提出多模态推理模型 Skywork R1V2,通过融合 MPO 与 GRPO 的混合强化学习及 SSB 机制提升推理与泛化,并在多项基准上达到开源领先水平。

摘要:

我们提出了 Skywork R1V2,这是一个新一代多模态推理模型,也是其前代 Skywork R1V 的一次重大飞跃。R1V2 的核心是引入了一种混合强化学习范式,联合利用 Mixed Preference Optimization(MPO,混合偏好优化)和 Group Relative Policy Optimization(GRPO,组相对策略优化),将奖励模型指导与基于规则的策略相协调,从而解决长期存在的在复杂推理能力与广泛泛化之间取得平衡的挑战。为进一步提升训练效率,我们提出 Selective Sample Buffer(SSB,选择性样本缓冲)机制,通过在整个优化过程中优先处理高价值样本,有效解决了 GRPO 固有的优势消失困境。值得注意的是,我们观察到过强的强化信号会诱发视觉幻觉——我们在整个训练过程中通过校准的奖励阈值对这一现象进行系统监测和缓解。实验结果证实了 R1V2 的卓越能力,其在多项基准上取得领先性能,例如 OlympiadBench 为 62.6、AIME2024 为 78.9、LiveCodeBench 为 63.6、MMMU 为 73.6。这些结果凸显了 R1V2 相对于现有开源模型的优越性,并表明其在缩小与包括 Gemini 2.5 和 OpenAI-o4-mini 在内的顶级专有系统的性能差距方面取得了显著进展。为促进开放性和可复现性,Skywork R1V2 模型权重已公开发布:https://huggingface.co/Skywork/Skywork-R1V2-38B。

英文摘要:

We present Skywork R1V2, a next-generation multimodal reasoning model and a major leap forward from its predecessor, Skywork R1V. At its core, R1V2 introduces a hybrid reinforcement learning paradigm that jointly leverages the Mixed Preference Optimization (MPO) and the Group Relative Policy Optimization (GRPO), which harmonizes reward-model guidance with rule-based strategies, thereby addressing the long-standing challenge of balancing sophisticated reasoning capabilities with broad generalization. To further enhance training efficiency, we propose the Selective Sample Buffer (SSB) mechanism, which effectively addresses the vanishing advantages dilemma inherent in GRPO by prioritizing high-value samples throughout the optimization process. Notably, we observe that excessive reinforcement signals can induce visual hallucinations--a phenomenon we systematically monitor and mitigate through calibrated reward thresholds throughout the training process. Empirical results affirm the exceptional capability of R1V2, with benchmark-leading performances such as 62.6 on OlympiadBench, 78.9 on AIME2024, 63.6 on LiveCodeBench, and 73.6 on MMMU. These results underscore R1V2's superiority over existing open-source models and demonstrate significant progress in closing the performance gap with premier proprietary systems, including Gemini 2.5 and OpenAI-o4-mini. The Skywork R1V2 model weights have been publicly released to promote openness and reproducibility https://huggingface.co/Skywork/Skywork-R1V2-38B.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑