arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.02143cs.LGcs.AIcs.CL
收藏

学习最佳N的生成选择

Learning Generative Selection for Best-of-N

Shubham Toshniwal, Aleksander Ficek, Siddhartha Jain, Wei Du, Vahid Noroozi, Sadegh Mahdavi, Somshubra Majumdar, Igor Gitman

AI总结:

通过强化学习,小模型可获得强大的生成选择能力,从而在推理任务中超越基线方法并接近大模型性能。

摘要:

通过并行采样扩展测试时间计算可以显著提高大语言模型的推理能力,但通常受限于最佳N选择的质量。生成选择方法,如GenSelect,解决了这一瓶颈,但强大的选择性能仍然主要受限于大模型。我们表明,通过针对性的强化学习,小推理模型可以获取强大的GenSelect能力。为此,我们从大规模数学和代码指令数据集中合成选择任务,通过过滤出同时具有正确和错误候选解决方案的实例,并训练17亿参数模型使用DAPO来奖励正确选择。在数学(AIME24,AIME25,HMMT25)和代码(LiveCodeBench)推理基准测试中,我们的模型在提示和多数投票基线方面表现一致,经常接近或超过更大的模型。此外,这些增益可以泛化到从更强的模型中选择输出,尽管训练只在较弱的模型输出上进行。总体而言,我们的结果证明强化学习是解锁小模型中强大生成选择的可扩展方法,使测试时间扩展更加高效。

英文摘要:

Scaling test-time compute via parallel sampling can substantially improve LLM reasoning, but is often limited by Best-of-N selection quality. Generative selection methods, such as GenSelect, address this bottleneck, yet strong selection performance remains largely limited to large models. We show that small reasoning models can acquire strong GenSelect capabilities through targeted reinforcement learning. To this end, we synthesize selection tasks from large-scale math and code instruction datasets by filtering to instances with both correct and incorrect candidate solutions, and train 1.7B-parameter models with DAPO to reward correct selections. Across math (AIME24, AIME25, HMMT25) and code (LiveCodeBench) reasoning benchmarks, our models consistently outperform prompting and majority-voting baselines, often approaching or exceeding much larger models. Moreover, these gains generalize to selecting outputs from stronger models despite training only on outputs from weaker models. Overall, our results establish reinforcement learning as a scalable way to unlock strong generative selection in small models, enabling efficient test-time scaling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑