CriticSearch：通过回顾性评论家实现搜索智能体的细粒度信用分配
CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic
AI总结：
针对现有搜索智能体强化学习优化存在的稀疏奖励问题，提出CriticSearch框架，通过回顾性评论家机制提供回合级密集反馈，在多跳推理基准上实现更快收敛、更稳定训练和更优性能。
摘要：
结合搜索引擎的工具集成推理（TIR）使大型语言模型能够迭代检索最新外部知识，提升复杂问答任务中的适应性和泛化能力。但现有搜索智能体流水线通常依赖基于强化学习的优化，常受稀疏结果奖励影响，导致探索低效和训练不稳定。我们提出CriticSearch，这是一个细粒度信用分配框架，通过回顾性评论家机制提供密集的回合级反馈。训练期间，冻结的非对称评论LLM利用完整轨迹和正确答案的特权信息回顾评估每个回合，将评估转化为稳定、密集的奖励以指导策略改进。在多种多跳推理基准上的实验结果表明，CriticSearch持续优于现有基线，实现更快收敛、更高训练稳定性和更好性能。
英文摘要：
Tool-Integrated Reasoning (TIR) with search engines enables large language models to iteratively retrieve up-to-date external knowledge, enhancing adaptability and generalization in complex question-answering tasks. However, existing search agent pipelines typically depend on reinforcement learning based optimization, which often suffers from sparse outcome rewards, leading to inefficient exploration and unstable training. We introduce CriticSearch, a fine-grained credit-assignment framework that supplies dense, turn-level feedback via a retrospective critic mechanism. During training, a frozen, asymmetric critique LLM retrospectively evaluates each turn using privileged information from the full trajectory and gold answers, converting these assessments into stable, dense rewards that guide policy improvement. Experimental results across diverse multi-hop reasoning benchmarks demonstrate that CriticSearch consistently outperforms existing baselines, achieving faster convergence, improved training stability, and higher performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
