COSMO-RL:通过联合安全性与稳定性实现可信赖的大型多模态推理模型
COSMO-RL: Towards Trustworthy LMRMs via Joint Safety and Stability
AI总结:
本文提出COSMO-RL混合强化学习框架,通过多模态多任务多目标训练,使LMRMs的安全性与能力共同提升,增强抗越狱鲁棒性并减少过度拒绝。
摘要:
大型多模态推理模型(LMRMs)正进入实际应用领域,在此类应用中,它们必须既实用又安全。安全性在多模态环境中尤其具有挑战性:图像和文本可以组合以绕过安全护栏,而单一目标的训练可能导致策略漂移,从而在良性输入上产生过度拒绝,或在风险输入上产生不安全的顺从。我们提出了COSMO-RL,一个混合强化学习框架,该框架在多模态、多任务和多目标信号下训练面向推理的LMRMs,并发布了由此产生的模型COSMO-R1。我们的方法旨在让安全性和能力在一个稳定的流程中共同增长,而不是在对齐过程中相互竞争。在实验中,COSMO-R1在保持——且通常提升——多模态推理和指令跟随能力的同时提高了安全性,展现出对多模态越狱攻击更强的鲁棒性,并减少了不必要的拒绝。该框架还能在不同骨干网络上迁移并带来一致的性能提升。消融研究支持了我们的设计选择,表明了一条在LMRMs中同时推进安全性和通用能力的简单路径。
英文摘要:
Large Multimodal Reasoning Models (LMRMs) are moving into real applications, where they must be both useful and safe. Safety is especially challenging in multimodal settings: images and text can be combined to bypass guardrails, and single objective training can cause policy drift that yields over-refusal on benign inputs or unsafe compliance on risky ones. We present COSMO-RL, a mixed reinforcement learning framework that trains reasoning oriented LMRMs under multimodal, multitask, and multiobjective signals, and we release the resulting model, COSMO-R1. Our approach aims to let safety and capability grow together in one stable pipeline rather than competing during alignment. In experiments, COSMO-R1 improves safety while maintaining-and often improving multimodal reasoning and instruction following, shows stronger robustness to multimodal jailbreaks, and reduces unnecessary refusals. The framework also transfers across backbones with consistent gains. Ablations support the design choices, indicating a simple path to advancing safety and general capability together in LMRMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。