CLIPO: 在策略优化中通过对比学习推广RLVR
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
AI总结：
CLIPO通过对比学习机制改进RLVR，提升大语言模型的策略优化泛化与鲁棒性
摘要：
可验证奖励的强化学习（RLVR）显著提高了大语言模型（LLMs）的推理能力。然而，RLVR仅依赖最终答案作为结果奖励，忽略了中间推理步骤的正确性。在这些过程错误但结果正确的轨迹上进行训练可能导致幻觉和答案复制，严重损害模型的泛化能力和鲁棒性。为了解决这一问题，我们将在策略优化（CLIPO）中引入对比学习机制，以推广RLVR过程。通过在成功的轨迹上优化对比损失，CLIPO引导LLM捕捉正确推理路径中共享的不变结构。这比RLVR中原有的单路径监督提供了更稳健的跨轨迹正则化，有效缓解了步骤级推理不一致，并抑制了幻觉特征。在实验中，CLIPO在多样化的推理基准上一致改进了多个RLVR基线，证明了对于LLM策略优化的泛化能力和鲁棒性有统一的提升。我们的代码和训练配方可在https://github.com/Qwen-Applications/CLIPO上获得。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
