arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.01248cs.LG
收藏

$S^3$-R1: 通过合成数据学习逐步检索与回答

$S^3$-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data

The University of Texas at Austin(德克萨斯大学奥斯汀分校);Google DeepMind(谷歌DeepMind)

Harsh Goel, Akhil Udathu, Susmija Jabbireddy, Pradnesh Kalkar, Atharva Parulekar

AI总结:

提出S^3-R1框架,通过合成数据生成和密集奖励信号,解决强化学习后训练中稀疏奖励和缺乏多跳问题数据的问题,提升模型搜索与问答能力。

摘要:

强化学习后训练使模型具备了新的能力,例如使用工具进行搜索。然而,这些模型主要面临两个限制:稀疏的基于结果的奖励,以及缺乏涵盖不同难度问题的训练数据,导致模型无法使用工具进行更深层次的搜索来收集证据以回答问题。为了解决这些限制,我们引入了S^3-R1(合成数据与稳定搜索R1),一个将数据驱动方法与更密集的学习信号相结合的框架。我们首先开发了一个合成生成与筛选流程,从现有文档中程序化地派生出多样化的多跳问题。该流程包含一个基于检索的验证步骤,专门用于分离出中等难度的问题。然后,我们将这个扩展的训练集与一个奖励结构配对,该结构同时评估中间搜索质量和最终答案的正确性。这种设置直接缓解了稀疏奖励固有的信用分配问题。我们的评估表明,S^3-R1通过学习更有效的搜索和综合策略,优于现有基线,在域外数据集上实现了高达10%的稳健泛化改进。

英文摘要:

Reinforcement learning (RL) post-training has enabled newer capabilities in models, such as agentic tool-use for search. However, these models struggle primarily due to limitations with sparse outcome-based rewards and a lack of training data that encapsulates questions of differing hardness, which results in models not performing deeper searches with tools to collect evidence for question-answering. To address these limitations, we introduce S^3-R1 (Synthetic data and stabilized Search R1), a framework that couples a data-centric approach with denser learning signals. We first develop a synthetic generation and curation pipeline that programmatically derives diverse, multi-hop questions from existing documents. This pipeline incorporates a retrieval-based verification step to specifically isolate questions of intermediate difficulty. We then pair this expanded training set with a reward structure that evaluates both intermediate search quality and the correctness of the final answer. This setup directly mitigates the credit assignment problems inherent to sparse rewards. Our evaluations show that S^3-R1 outperforms existing baselines by learning more effective search and synthesis strategies, yielding up to a 10% improvement in robust generalization on out-of-domain datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑