从跨领域视角重新审视用于大语言模型推理的强化学习
Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
AI总结：
本文提出跨领域RL语料库Guru，系统研究LLM推理的强化学习，发现预训练暴露少的领域需领域内训练，并发布Guru-7B/32B模型，在17任务上超越基线7.9%和6.7%。
摘要：
强化学习（RL）已成为提升大语言模型（LLM）推理能力的一种有前景的方法，然而大多数开源工作仅聚焦于数学和代码领域，限制了我们对其在通用推理中更广泛适用性的理解。一个关键挑战在于，在多样化的推理领域中缺乏可靠且可扩展的RL奖励信号。我们引入了Guru，一个精心策划的RL推理语料库，包含92K个可验证示例，涵盖六个推理领域——数学、代码、科学、逻辑、模拟和表格——每个领域都通过特定领域的奖励设计、去重和过滤来确保RL训练的可靠性和有效性。基于Guru，我们系统地重新审视了RL用于LLM推理的既有发现，并观察到跨领域的显著差异。例如，先前的研究表明RL主要从预训练模型中激发已有知识，而我们的结果揭示了更微妙的模式：预训练期间频繁出现的领域（数学、代码、科学）容易从跨领域RL训练中受益，而预训练暴露有限的领域（逻辑、模拟和表格）则需要领域内训练才能获得有意义的性能提升，这表明RL可能促进真正的技能习得。最后，我们提出了Guru-7B和Guru-32B两个模型，它们在公开数据上经过RL训练的开源模型中达到了最先进的性能，在我们涵盖六个推理领域的17任务评估套件上，分别比最佳基线高出7.9%和6.7%。我们还表明，我们的模型有效提升了其基础模型的Pass@k性能，尤其是在预训练数据中不太可能出现的复杂任务上。我们发布了数据、模型、训练和评估代码，以促进通用推理，网址为：https://github.com/LLM360/Reasoning360
英文摘要：
Reinforcement learning (RL) has emerged as a promising approach to improve large language model (LLM) reasoning, yet most open efforts focus narrowly on math and code, limiting our understanding of its broader applicability to general reasoning. A key challenge lies in the lack of reliable, scalable RL reward signals across diverse reasoning domains. We introduce Guru, a curated RL reasoning corpus of 92K verifiable examples spanning six reasoning domains--Math, Code, Science, Logic, Simulation, and Tabular--each built through domain-specific reward design, deduplication, and filtering to ensure reliability and effectiveness for RL training. Based on Guru, we systematically revisit established findings in RL for LLM reasoning and observe significant variation across domains. For example, while prior work suggests that RL primarily elicits existing knowledge from pretrained models, our results reveal a more nuanced pattern: domains frequently seen during pretraining (Math, Code, Science) easily benefit from cross-domain RL training, while domains with limited pretraining exposure (Logic, Simulation, and Tabular) require in-domain training to achieve meaningful performance gains, suggesting that RL is likely to facilitate genuine skill acquisition. Finally, we present Guru-7B and Guru-32B, two models that achieve state-of-the-art performance among open models RL-trained with publicly available data, outperforming best baselines by 7.9% and 6.7% on our 17-task evaluation suite across six reasoning domains. We also show that our models effectively improve the Pass@k performance of their base models, particularly on complex tasks less likely to appear in pretraining data. We release data, models, training and evaluation code to facilitate general-purpose reasoning at: https://github.com/LLM360/Reasoning360
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
