arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.23440cs.CLcs.IR
收藏

截断步骤级采样与过程奖励用于检索增强推理

Truncated Step-Level Sampling with Process Rewards for Retrieval-Augmented Reasoning

Chris Samarinas, Haw-Shiuan Chang, Hamed Zamani

AI总结:

本文提出SLATE方法,通过截断步骤级采样和密集过程奖励解决检索增强推理中的信用分配问题,实验表明其在多跳任务中表现优异。

摘要:

强化学习已成为训练大语言模型以交织推理与搜索引擎调用的有效范式。然而,现有方法面临根本的信用分配问题:像Search-R1这样的方法将单一结果奖励分配给整个多步骤轨迹,无法提供关于哪些推理或检索决策导致成功或失败的信号。过程奖励方法如StepSearch引入了步骤级监督,但仍独立采样完整轨迹,因此任意给定步骤的优势估计受其他步骤随机性污染。我们提出SLATE(截断探索的步骤级优势估计),通过两种互补思想解决这两个问题。首先,截断步骤级采样从共享前缀生成k个延续,将所有变化限制到单个决策点。我们证明这将优势估计的方差降低至完整轨迹采样的因数T,这是检索增强推理中步骤级RL的第一个正式方差保证。其次,密集、分解的过程奖励通过LLM裁判器分别评估推理质量、查询质量和答案正确性,通过三元尺度提供比二元结果信号或启发式步骤级分数更丰富的监督。在七个问答基准测试中,SLATE在稀疏奖励和过程奖励基线中表现一致,实现7B模型7.0%和3B模型30.7%的相对提升。增益最大出现在具有挑战性的多跳任务中,消融实验确认截断采样和密集奖励提供互补的益处。

英文摘要:

Reinforcement learning has emerged as an effective paradigm for training large language models to interleave reasoning with search engine calls. However, existing approaches face a fundamental credit assignment problem: methods like Search-R1 assign a single outcome reward to the entire multi-step trajectory, providing no signal about which reasoning or retrieval decisions were responsible for success or failure. Process-reward methods such as StepSearch introduce step-level supervision but still sample complete trajectories independently, so advantage estimates at any given step are contaminated by the randomness of all other steps. We propose SLATE (Step-Level Advantage estimation for Truncated Exploration), which addresses both problems through two complementary ideas. First, truncated step-level sampling generates k continuations from a shared prefix, isolating all variation to a single decision point. We prove this reduces the variance of advantage estimates by up to a factor of T compared to full-trajectory sampling for T-step trajectories, the first formal variance guarantee for step-level RL in retrieval-augmented reasoning. Second, dense, decomposed process rewards separately evaluate reasoning quality, query quality, and answer correctness on a ternary scale via an LLM judge, providing richer supervision than binary outcome signals or heuristic step-level scores. Experiments on seven QA benchmarks show that SLATE consistently outperforms both sparse-reward and process-reward baselines, achieving a 7.0% relative improvement over Search-R1 on the 7B model and 30.7% on the 3B model. Gains are largest on challenging multi-hop tasks, and ablations confirm that truncated sampling and dense rewards provide complementary benefits.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑