arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.22975cs.AI
收藏

金 Goose:从不可验证的互联网文本中合成无限 RLVR 任务的简单技巧

Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text

NVIDIA(NVIDIA公司);University of Washington(华盛顿大学);University of California San Diego(圣地亚哥大学)
Ximing Lu, David Acuna, Jaehun Jung, Jian Hu, Di Zhang, Shizhe Diao, Yunheng Zou, Shaokun Zhang, Brandon Cui, Mingjie Liu, Hyunwoo Kim, Prithviraj Ammanabrolu, … 展开作者

Ximing Lu, David Acuna, Jaehun Jung, Jian Hu, Di Zhang, Shizhe Diao, Yunheng Zou, Shaokun Zhang, Brandon Cui, Mingjie Liu, Hyunwoo Kim, Prithviraj Ammanabrolu, Jan Kautz, Yi Dong, Yejin Choi

AI总结:

Golden Goose 通过从不可验证的互联网文本中合成无限 RLVR 任务,提升大型语言模型在复杂推理和网络安全领域的性能。

摘要:

可验证奖励的强化学习(RLVR)已成为解锁大型语言模型(LLMs)复杂推理能力的基石。然而,扩大 RL 的瓶颈在于现有可验证数据的有限性,其中改进在长期训练中趋于饱和。为此,我们提出了 Golden Goose,一种简单的技巧,通过构造填空任务的多选问答版本,从不可验证的互联网文本中合成无限 RLVR 任务。给定源文本,我们提示 LLM 识别并遮蔽关键推理步骤,然后生成一组多样且合理的干扰项。这使我们能够利用通常不包含在先前 RLVR 数据构建中的富含推理的不可验证语料库(例如科学教科书)来合成 GooseReason-0.7M,一个包含超过 0.7 百万任务的大型 RLVR 数据集,涵盖数学、编程和一般科学领域。实证表明,GooseReason 有效恢复了在现有 RLVR 数据上饱和的模型,使其在连续 RL 下获得稳健且持续的提升,并在 15 个多样化的基准测试中实现新的最先进的结果。最后,我们将在现实世界中部署 Golden Goose,从原始 FineWeb 抓取数据为网络安全领域合成 RLVR 任务,其中不存在先前的 RLVR 数据。在 GoWen3-4B-Instruct 上训练所得数据 GooseReason-Cyber 在网络安全领域实现了新的最先进结果,超越了一个经过广泛领域特定预训练和后训练的 7B 领域专用模型。这突显了通过利用丰富推理的大量不可验证互联网文本自动扩展 RLVR 数据的潜力。

英文摘要:

Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for unlocking complex reasoning in Large Language Models (LLMs). Yet, scaling up RL is bottlenecked by limited existing verifiable data, where improvements increasingly saturate over prolonged training. To overcome this, we propose Golden Goose, a simple trick to synthesize unlimited RLVR tasks from unverifiable internet text by constructing a multiple-choice question-answering version of the fill-in-the-middle task. Given a source text, we prompt an LLM to identify and mask key reasoning steps, then generate a set of diverse, plausible distractors. This enables us to leverage reasoning-rich unverifiable corpora typically excluded from prior RLVR data construction (e.g., science textbooks) to synthesize GooseReason-0.7M, a large-scale RLVR dataset with over 0.7 million tasks spanning mathematics, programming, and general scientific domains. Empirically, GooseReason effectively revives models saturated on existing RLVR data, yielding robust, sustained gains under continuous RL and achieving new state-of-the-art results for 1.5B and 4B-Instruct models across 15 diverse benchmarks. Finally, we deploy Golden Goose in a real-world setting, synthesizing RLVR tasks from raw FineWeb scrapes for the cybersecurity domain, where no prior RLVR data exists. Training Qwen3-4B-Instruct on the resulting data GooseReason-Cyber sets a new state-of-the-art in cybersecurity, surpassing a 7B domain-specialized model with extensive domain-specific pre-training and post-training. This highlights the potential of automatically scaling up RLVR data by exploiting abundant, reasoning-rich, unverifiable internet text.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑