重新思考用于RLVR的多项选择题：通过干扰项设计解锁潜力
Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design
AI总结：
本文研究了干扰项设计对RLVR的影响，发现选项数量不匹配和强干扰项能有效减少随机猜测，提出IDC框架提升干扰项质量，实验表明在多个基准上显著提升了RLVR训练效果。
摘要：
强化学习与可验证奖励（RLVR）显著增强了大语言模型的推理能力。当应用于RLVR时，多项选择题（MCQs）提供了一个可扩展的可验证数据源，但存在诱导奖励黑客的风险，即模型通过随机猜测或简单排除来捷径推理。当前方法通常通过将MCQs转换为开放式格式来缓解这一问题，从而丢弃了专家设计的干扰项提供的对比信号。在本文中，我们系统地研究了选项设计对RLVR的影响。我们的分析得出两个主要见解：（1）训练和测试之间选项数量不匹配会降低性能。（2）强干扰项有效缓解随机猜测，即使在2选1的问题中也能促进有效的RLVR训练。受这些发现的启发，我们提出了迭代干扰项校准（IDC），一个主动构建高质量干扰项的框架，以阻止排除捷径并促进深入推理。在各种基准上的实验表明，我们的方法有效提升了干扰项质量，并在RLVR训练中相比原始数据产生了显著的提升。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capabilities of Large Language Models. When applied to RLVR, Multiple-Choice Questions (MCQs) offer a scalable source of verifiable data but risk inducing reward hacking, where models shortcut reasoning via random guessing or simple elimination. Current approaches often mitigate this by converting MCQs to open-ended formats, thereby discarding the contrastive signal provided by expert-designed distractors. In this work, we systematically investigate the impact of option design on RLVR. Our analysis highlights two primary insights: (1) Mismatches in option counts between training and testing degrade performance. (2) Strong distractors effectively mitigate random guessing, enabling effective RLVR training even with 2-way questions. Motivated by these findings, we propose Iterative Distractor Curation (IDC), a framework that actively constructs high-quality distractors to block elimination shortcuts and promote deep reasoning. Experiments on various benchmarks demonstrate that our method effectively enhances distractor quality and yields significant gains in RLVR training compared to the original data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
