RICE-PO：将检索交互转化为推理智能体的信用信号
RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents
AI总结：
提出RICE-PO框架，通过将检索交互转化为局部学习信号，解决推理型检索智能体在训练中推理步骤的信用分配问题，在BRIGHT和BEIR上优于基线。
摘要：
检索正从一次性匹配向交互式推理转变，语言智能体迭代检查证据、重新表述查询并再次搜索。训练此类智能体面临信用分配挑战：可执行动作（如查询或摘要）可由检索器直接评估，而潜在推理步骤不可直接观察，仅影响未来的可执行动作。这种不对称使得结果级奖励分配不可靠，因为相同的最终奖励可能奖励那些实际上并未促成检索成功的推理步骤。我们提出RICE-PO，一种无批评策略优化框架，将检索交互转化为局部学习信号。RICE-PO选择高不确定性的可执行动作作为锚点，使用检索指标评估局部反事实分支，并仅在推理到动作的影响强且未来残差效应稳定时，将信用传播给潜在推理步骤。在BRIGHT和BEIR上，RICE-PO在相同检索器设置下始终优于基于提示的智能体和基于组的强化学习基线。这些结果表明，智能体-环境交互结构本身可以为训练基于推理的检索智能体提供有用的监督。
英文摘要：
Retrieval is increasingly moving from one-shot matching toward interactive reasoning, where language agents iteratively inspect evidence, reformulate queries, and search again. Training such agents raises a credit-assignment challenge: executable actions such as queries or summaries can be directly evaluated by the retriever, while latent reasoning steps are not directly observable and only affect future executable actions. This asymmetry makes outcome-level reward assignment unreliable, as the same final reward may credit reasoning steps that did not actually shape retrieval success. We propose RICE-PO, a critic-free policy optimization framework that converts retrieval interactions into localized learning signals. RICE-PO selects high-uncertainty executable actions as anchors, evaluates local counterfactual branches using retrieval metrics, and propagates credit to latent reasoning steps only when reasoning-to-action influence is strong and future residual effects are stable. On BRIGHT and BEIR, RICE-PO consistently outperforms prompt-based agents and group-based RL baselines under the same retriever setting. These results show that the structure of agent-environment interaction itself can provide useful supervision for training reasoning-based retrieval agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
