arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.20197cs.LGcs.AI
收藏

混合策略强化学习可调节探索用于多模态推理

Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning

Aberystwyth University(阿伯里斯特维斯大学);Institute of Artificial Intelligence (TeleAl)(人工智能研究所);China Telecom(中国电信);Tsinghua University(清华大学)

Zhuoxu Huang, Mengxi Jia, Hao Sun, Xuelong Li, Jungong Han

AI总结:

本文提出CalibRL框架,通过分布感知优势加权和非对称激活函数实现可控探索,提升多模态推理模型的探索与利用平衡。

摘要:

可验证奖励的强化学习(RLVR)已成为增强多模态大语言模型(MLLM)推理能力的主要学习范式。然而,在RL训练过程中,MLLM巨大的状态空间和稀疏奖励往往导致熵崩溃、策略退化或过度利用次优行为。这需要一种探索策略,能够在保持有效随机性的同时避免不受控制的随机采样所带来的低效探索。在本文中,我们提出CalibRL,一种支持受专家指导的可调节探索的混合策略RLVR框架,通过两个关键机制实现。首先,分布感知的优势加权通过群体稀有性来调整更新,从而校准分布,因此保持探索。同时,非对称激活函数(LeakyReLU)利用专家知识作为校准基线,以调节过度自信的更新,同时保持其纠正方向。CalibRL通过在线采样估计在线策略分布,以指导的方式增加策略熵,并通过估计在线策略分布来明确目标分布。更新由这些信息行为驱动,避免收敛到错误模式。重要的是,这些设计有助于缓解模型策略与专家轨迹之间的分布不匹配,从而在探索和利用之间实现更稳定的平衡。在八个基准测试中,包括域内和域外设置,广泛实验展示了持续改进,验证了我们可控混合策略RLVR训练的有效性。代码可在https://github.com/zhh6425/CalibRL上获得。

英文摘要:

Reinforcement Learning with verifiable rewards (RLVR) has emerged as a primary learning paradigm for enhancing the reasoning capabilities of multi-modal large language models (MLLMs). However, during RL training, the enormous state space of MLLM and sparse rewards often leads to entropy collapse, policy degradation, or over-exploitation of suboptimal behaviors. This necessitates an exploration strategy that maintains productive stochasticity while avoiding the drawbacks of uncontrolled random sampling, yielding inefficient exploration. In this paper, we propose CalibRL, a hybrid-policy RLVR framework that supports controllable exploration with expert guidance, enabled by two key mechanisms. First, a distribution-aware advantage weighting scales updates by group rareness to calibrate the distribution, therefore preserving exploration. Meanwhile, the asymmetric activation function (LeakyReLU) leverages the expert knowledge as a calibration baseline to moderate overconfident updates while preserving their corrective direction. CalibRL increases policy entropy in a guided manner and clarifies the target distribution by estimating the on-policy distribution through online sampling. Updates are driven by these informative behaviors, avoiding convergence to erroneous patterns. Importantly, these designs help alleviate the distributional mismatch between the model's policy and expert trajectories, thereby achieving a more stable balance between exploration and exploitation. Extensive experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements, validating the effectiveness of our controllable hybrid-policy RLVR training. Code is available at https://github.com/zhh6425/CalibRL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑