通过可验证的多选改写扩展RLVR以应对开放性任务
Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
AI总结：
本文提出VMR-RLVR方法，通过将开放性任务数据转化为可验证的多选格式，提升LLM在无明确真实值情况下开放性任务的推理能力。
摘要：
可验证的强化学习奖励（RLVR）在增强大语言模型（LLMs）的推理能力方面展现了巨大潜力。然而，其成功迄今为止主要局限于数学和编程领域，这些领域具有明确且可自动验证的结果。在开放性任务（如创造性写作和主观问答）上进行强化学习仍然依赖于奖励模型，因为缺乏可验证的解决方案。这引发了一个关键问题：如何扩展RLVR以在没有明确真实值的情况下加强开放性任务的推理能力？为克服这一挑战，我们引入了可验证的多选改写强化学习（VMR-RLVR），一种新的训练策略，将开放性数据改写成可验证的多选格式，即使在没有显式真实值的情况下也能有效训练。在多个基准上的实验结果验证了我们的方法在提高LLM在开放性任务上的性能方面的有效性。值得注意的是，在七个开放性基准上，我们的VMR-RLVR训练在RL与奖励模型的基础上平均提高了3.29分。
英文摘要：
Reinforcement Learning with Verifiable Rewards(RLVR) has demonstrated great potential in enhancing the reasoning capabilities of large language models (LLMs). However, its success has thus far been largely confined to the mathematical and programming domains with clear and automatically checkable outcomes. Reinforcement learning on open-ended tasks (e.g., creative writing and subjective Q&A) continues to rely on reward models due to the absence of verifiable solutions. This raises a key question: how can we extend RLVR to strengthen reasoning in open-ended tasks regardless of the absence of the unambiguous ground truth? To overcome this challenge, we introduce Verifiable Multiple-Choice Reformulation for Reinforcement Learning from Verifiable Rewards (VMR-RLVR), a novel training strategy that restructures open-ended data into verifiable multiple-choice formats, enabling effective training even in the absence of explicit ground truth. Experimental results on multiple benchmarks validate the effectiveness of our method in improving LLM performance on open-ended tasks. Notably, across seven open-ended benchmarks, our VMR-RLVR training delivers an average gain of 3.29 points over the RL with reward model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
