arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.20117cs.AIcs.LG
收藏

ReSyn:自主扩展合成环境以增强推理模型

ReSyn: Autonomously Scaling Synthetic Environments for Reasoning Models

Carnegie Mellon University(卡内基梅隆大学);Amazon Web Services(亚马逊网络服务)

Andre He, Nathaniel Weir, Kaj Bostrom, Allen Nie, Darion Cassel, Sam Bayless, Huzefa Rangwala

AI总结:

ReSyn通过生成多样化推理环境,利用验证器监督和任务多样性提升推理语言模型的推理能力。

摘要:

可验证奖励的强化学习(RLVR)已显现为一种有前途的方法,用于通过利用验证器的监督来训练推理语言模型(RLMs)。尽管验证器的实现比解决方案注释更容易,但现有合成数据生成方法仍然主要以解决方案为中心,而基于验证器的方法依赖于少量手工制作的程序环境。在本文中,我们通过引入ReSyn,一个生成多样推理环境的管道,扩展RLVR。该管道配备了实例生成器和验证器,涵盖约束满足、算法谜题和空间推理等任务。一个在ReSyn数据上通过强化学习训练的Qwen2.5-7B-Instruct模型在推理基准和跨领域数学基准上实现了持续的提升,包括在具有挑战性的BBEH基准上实现了27%的相对提升。消融研究显示,基于验证器的监督和增加的任务多样性都显著贡献,提供了实证证据,表明在大规模生成推理环境可以增强RLMs的推理能力。

英文摘要:

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising approach for training reasoning language models (RLMs) by leveraging supervision from verifiers. Although verifier implementation is easier than solution annotation for many tasks, existing synthetic data generation methods remain largely solution-centric, while verifier-based methods rely on a few hand-crafted procedural environments. In this work, we scale RLVR by introducing ReSyn, a pipeline that generates diverse reasoning environments equipped with instance generators and verifiers, covering tasks such as constraint satisfaction, algorithmic puzzles, and spatial reasoning. A Qwen2.5-7B-Instruct model trained with RL on ReSyn data achieves consistent gains across reasoning benchmarks and out-of-domain math benchmarks, including a 27\% relative improvement on the challenging BBEH benchmark. Ablations show that verifier-based supervision and increased task diversity both contribute significantly, providing empirical evidence that generating reasoning environments at scale can enhance reasoning abilities in RLMs

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑