增强多样性推理以应对主观问题
Diversity-Enhanced Reasoning for Subjective Questions
AI总结：
本文提出MultiRole-R1框架，通过引入视角多样性和token级多样性提升主观推理性能，实验显示其在主观任务和高级数学推理中均取得显著提升。
摘要：
大型推理模型（LRMs）具有长链推理能力，通过强化学习与可验证奖励（RLVR）进行优化，在客观推理任务如数学问题解决和代码生成方面表现优异。然而，RLVR以降生成多样性而闻名，这导致LRMs在需要多种答案的主观推理任务中表现不佳。尽管近期研究认识到在客观推理中增强多样性训练的重要性，但对主观任务的关注有限。在本文中，我们发现通过引入视角多样性和token级多样性可以提升主观推理，前者为现实世界利益相关者群体提供连贯的支撑结构，后者则扩大了答案搜索空间。我们提出了MultiRole-R1，一个增强多样性的训练框架，其包含一个无监督的数据构建管道，合成包含各种角色视角的推理链。它还采用强化学习通过群体相对策略优化与奖励塑造，将多样性作为奖励信号，除了可验证奖励。在仅训练主观任务的情况下，MultiRole-R1将领域内和领域外的准确性分别提高了14.1%和7.64%，甚至在高级数学推理如AIME 2024上也提升了性能。我们进一步表明，多样性比推理长度是更一致的准确性指标。
英文摘要：
Large Reasoning Models (LRMs) with long chain-of-thought capabilities, optimized via reinforcement learning with verifiable rewards (RLVR), excel at objective reasoning tasks like mathematical problem solving and code generation. However, RLVR is known for degrading generation diversity, which causes LRMs to fall short on subjective reasoning that has multiple answers depending on different role perspectives. While recent studies recognize the importance of diversity-enhanced training in objective reasoning, limited attention has been given to subjective tasks. In this paper, we find that subjective reasoning can be improved by introducing perspective diversity and token-level diversity, with the former one providing a coherent scaffolding anchored to a real-world stakeholder group and the latter one broadening the answer search space. We propose MultiRole-R1, a diversity-enhanced training framework featuring an unsupervised data construction pipeline that synthesizes reasoning chains incorporating various role perspectives. It also employs reinforcement learning via Group Relative Policy Optimization with reward shaping, taking diversity as a reward signal in addition to verifiable reward. Training on subjective tasks solely, MultiRole-R1 increases the in-domain and out-of-domain accuracy by 14.1% and 7.64%, and even enhances the performance on advanced math reasoning such as AIME 2024. We further show that diversity is a more consistent indicator of accuracy than reasoning length.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
