arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.17877cs.AI
收藏

PAIR:面向多轮代理优化的前缀感知内部奖励模型

PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization

KAIST(韩国科学技术院);Yonsei University(延世大学)

Wonjoong Kim, Yeonjun In, Sangwu Park, Dongha Lee, Chanyoung Park

AI总结:

本文提出PAIR模型,通过结合冻结的隐藏状态探针和轻量级注意力头部,解决多轮任务中内部正确性探针的可靠性问题,从而在不依赖外部模型调用或地面真实依赖的情况下,为GRPO训练提供密集的步骤级奖励信号。

摘要:

当前LLM在执行复杂多阶段任务方面面临重大挑战。组相对策略优化(GRPO)已成为主流选择,但其依赖稀疏结果奖励严重限制了中间步骤的信用分配。现有解决方案如运行完整回滚以分配步骤级优势、在每个步骤调用外部LLM评判者或计算内在奖励(需要每次评估都有地面真实答案)都引入了显著成本或实际限制。我们假设内部正确性探针可以重新利用LLM隐藏状态进行步骤级奖励信号,可能一次性解决所有这些限制。然而,现有探针研究假设输入干净,我们首先表明在多步骤设置中这一假设不成立:隐藏状态探针在前缀污染跟踪与可能损坏的前缀保持一致性时严重退化,而基于注意力的特征在污染下保持稳健但清洁前缀表现欠佳。基于这种互补关系,我们提出前缀感知内部奖励(PAIR),一种两阶段模型,包含冻结隐藏状态探针估计信念一致性以及轻量级注意力头部纠正其向地面正确性。实验结果表明,PAIR在受污染轨迹上实现了最高的AUROC,同时运行成本极低,能够在不依赖外部模型调用、地面真实依赖或完整轨迹回滚的情况下,为GRPO训练提供密集的步骤级奖励信号。

英文摘要:

A significant hurdle for current LLMs is the execution of complex, multi-stage tasks. Group Relative Policy Optimization (GRPO) has been emerging as a leading choice, but its reliance on sparse outcome rewards severely limits credit assignment across intermediate steps. Existing remedies such as running full rollouts to assign step-level advantages, calling external LLM judges at each step, or computing intrinsic rewards that require ground-truth answers at every evaluation introduce significant costs or practical constraints. We hypothesize that internal correctness probing over LLM hidden states can be repurposed as a step-level reward signal, potentially addressing all of these limitations at once. However, existing probing research assumes clean inputs, and we first show that this assumption breaks down in multi-step settings: hidden-state probes degrade severely under prefix contamination tracking coherence with the (possibly corrupted) prefix rather than grounded correctness, while attention-based features remain robust to contamination but underperform on clean prefixes. Building on this complementary relationship, we propose the Prefix-Aware Internal Reward (PAIR), a two-stage model with a frozen hidden-state probe estimating belief-consistency and a lightweight attention-based head correcting it toward grounded correctness. Experimental results show that PAIR achieves the highest AUROC on contaminated trajectories while operating at negligible inference cost, enabling dense step-level reward signals for GRPO training without external model calls, ground-truth dependencies, or full-trajectory rollouts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑