SUPERNOVA: 通过自然指令上的强化学习激发大语言模型的通用推理
SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions
AI总结：
本文提出SUPERNOVA框架，通过自然指令数据集构建高质量的强化学习可验证奖励数据集，通过100+次强化学习实验系统研究如何利用这些数据集提升下游推理性能，并在BigBench Extra Hard基准上实现64.4个百分点的相对提升。
摘要：
强化学习可验证奖励（RLVR）在数学和代码等正式领域显著提升了推理能力，但将其扩展到STEM领域以外仍然具有挑战性。扩展RLVR到STEM领域本质上受到高质量可验证训练数据的缺乏限制。在本文中，我们引入SUPERNOVA，一个从自然指令数据集中整理RLVR数据的框架，这些数据集是专家标注的丰富来源，但尚未被充分利用于RLVR训练。通过100多次受控的强化学习实验，我们系统研究如何利用这些数据集进行RLVR训练以及数据整理决策如何影响下游推理性能。特别是，我们研究了三种数据设计：（a）源任务选择，（b）任务混合，以及（c）合成干预。我们的分析揭示了源任务选择对下游推理性能有显著影响。此外，基于单个目标任务性能选择任务优于基于总体平均性能的策略，合成干预并未提高推理能力。受这些见解的启发，我们构建了SUPERNOVA，一个从自然指令数据集中整理出的25,000个实例的高质量RLVR数据集。我们证明了在SUPERNOVA上训练Qwen3-0.6B比基础Qwen3-0.6B表现更优，在包含23个复杂推理任务的挑战性基准BigBench Extra Hard（BBEH）上实现了64.4个百分点的相对提升。重要的是，我们发现SUPERNOVA的收益可以推广到未见基准、更大模型规模和新模型家族。总体而言，我们的发现为整理人类标注资源以扩展RLVR到通用推理提供了实用见解。模型、数据、代码见https://github.com/asuvarna31/supernova。
英文摘要：
Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved reasoning in formal domains such as mathematics and code, but extending these gains beyond STEM remains challenging. Extending RLVR beyond STEM is fundamentally constrained by the lack of high-quality verifiable training data. In this work, we introduce SUPERNOVA, a framework for curating RLVR data from natural instruction datasets, which are a rich source of expert-annotated data but are underexplored for RLVR training. Through 100+ controlled RL experiments, we systematically study how to utilize these dataset for RLVR and how data curation decisions affect downstream reasoning performance . In particular, we investigate three data designs: (a) source task selection, (b) task mixing, and (c) synthetic interventions. Our analysis reveals that source task selection has a significant impact on downstream reasoning performance. Moreover, selecting tasks based on their performance for individual target tasks outperforms strategies based on overall average performance and synthetic interventions do not improve reasoning. Guided by these insights, we construct SUPERNOVA, a high-quality RLVR dataset of 25K instances curated from natural instruction datasets. We show that training Qwen3-0.6B on SUPERNOVA outperforms the base Qwen3-0.6B, yielding a relative gain of 64.4pp on BigBench Extra Hard (BBEH), a challenging benchmark comprising 23 complex reasoning tasks. Importantly, we find that gains from SUPERNOVA generalize to unseen benchmarks, larger model scales, and newer model families. Overall, our findings provide practical insights for curating human-annotated resources to extend RLVR to general reasoning. Models, Data, Code at https://github.com/asuvarna31/supernova.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
