arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.08282cs.CL
收藏

D$^2$Plan: 双智能体动态全局规划用于复杂检索增强推理

D$^2$Plan: Dual-Agent Dynamic Global Planning for Complex Retrieval-Augmented Reasoning

Peking University(北京大学)

Kangcheng Luo, Tinglang Wu, Yansong Feng

AI总结:

D$^2$Plan通过双智能体协作提升复杂检索增强推理的连贯性和抗干扰能力。

摘要:

近期经过强化学习(RL)训练的搜索增强型LLM能够将搜索与推理交织在一起以处理多跳推理任务。然而,随着上下文的积累,上下文被大量关键证据和无关信息淹没,导致两个关键的失败模式:(1)无效的搜索链构建,产生错误的查询或遗漏关键信息的检索;(2)外围证据导致推理劫持,使模型将干扰项误认为有效证据。为了解决这些挑战,我们提出了**D$^2$Plan**,一种用于复杂检索增强推理的**双智能体动态全局规划范式**。**D$^2$Plan**通过*推理者*和*净化器*的协作运作:*推理者*在推理过程中构建显式的全局计划,并根据检索反馈动态调整;*净化器*评估检索相关性,并为*推理者*浓缩关键信息。我们进一步引入了一个两阶段训练框架,包括对合成轨迹进行监督微调(SFT)冷启动和具有计划导向奖励的强化学习,以教导LLM掌握**D$^2$Plan**范式。广泛的实验表明,**D$^2$Plan**能够实现更连贯的多步推理和更强的抗无关信息能力,从而在具有挑战性的问答基准上取得更优的性能。

英文摘要:

Recent search-augmented LLMs trained with reinforcement learning (RL) can interleave searching and reasoning for multi-hop reasoning tasks. However, they face two critical failure modes as the accumulating context becomes flooded with both crucial evidence and irrelevant information: (1) ineffective search chain construction that produces incorrect queries or omits retrieval of critical information, and (2) reasoning hijacking by peripheral evidence that causes models to misidentify distractors as valid evidence. To address these challenges, we propose **D$^2$Plan**, a **D**ual-agent **D**ynamic global **Plan**ning paradigm for complex retrieval-augmented reasoning. **D$^2$Plan** operates through the collaboration of a *Reasoner* and a *Purifier*: the *Reasoner* constructs explicit global plans during reasoning and dynamically adapts them based on retrieval feedback; the *Purifier* assesses retrieval relevance and condenses key information for the *Reasoner*. We further introduce a two-stage training framework consisting of supervised fine-tuning (SFT) cold-start on synthesized trajectories and RL with plan-oriented rewards to teach LLMs to master the **D$^2$Plan** paradigm. Extensive experiments demonstrate that **D$^2$Plan** enables more coherent multi-step reasoning and stronger resilience to irrelevant information, thereby achieving superior performance on challenging QA benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑