Atom-Searcher：通过细粒度原子思维奖励增强智能体深度研究
Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward
AI总结：
针对智能体深度研究中结果强化学习的奖励稀疏与梯度冲突问题，提出 Atomic Thought 细粒度分解及 Atom-Searcher 课程奖励框架，在七个基准上实现一致提升。
摘要：
大型语言模型（LLM）展现出卓越的问题解决能力，但由于静态的内部知识，在处理复杂任务时存在困难。检索增强生成（RAG）增强了对外部信息的访问，但由于工作流僵化，在多跳推理和策略性搜索方面仍然受限。近期智能体深度研究的进展使 LLM 能够自主推理、搜索并综合信息。然而，当前依赖基于结果的强化学习（RL）的方法面临梯度冲突和奖励稀疏等关键问题，限制了性能提升和训练效率。为解决这些问题，我们首先提出 Atomic Thought（原子思维），这是一种新型 LLM 思维范式，将推理分解为细粒度的功能单元。这些单元由 Reasoning Reward Models（RRM，推理奖励模型）监督，后者提供 Atomic Thought Rewards（ATR，原子思维奖励）以进行细粒度引导。在此基础上，我们提出 Atom-Searcher，这是一种用于智能体深度研究的新型 RL 框架，整合了 Atomic Thought 与 ATR。Atom-Searcher 采用受课程学习启发的奖励调度，在早期优先使用过程级 ATR，随后过渡到结果奖励，从而加快在有效推理路径上的收敛。在七个基准测试上的实验表明，其相较最先进方法取得了一致提升。主要优势包括：（1）Atom-Searcher 可在测试时扩展计算；（2）Atomic Thought 为 RRM 提供监督锚点，连接深度研究任务与 RRM；（3）Atom-Searcher 展现出更具可解释性、更接近人类的推理模式。
英文摘要：
Large language models (LLMs) exhibit remarkable problem-solving abilities, but struggle with complex tasks due to static internal knowledge. Retrieval-Augmented Generation (RAG) enhances access to external information, yet remains limited in multi-hop reasoning and strategic search due to rigid workflows. Recent advancements in agentic deep research empower LLMs to autonomously reason, search, and synthesize information. However, current approaches relying on outcome-based reinforcement learning (RL) face critical issues such as conflicting gradients and reward sparsity, limiting performance gains and training efficiency. To address these, we first propose Atomic Thought, a novel LLM thinking paradigm that decomposes reasoning into fine-grained functional units. These units are supervised by Reasoning Reward Models (RRMs), which provide Atomic Thought Rewards (ATR) for fine-grained guidance. Building on this, we propose Atom-Searcher, a novel RL framework for agentic deep research that integrates Atomic Thought and ATR. Atom-Searcher uses a curriculum-inspired reward schedule, prioritizing process-level ATR early and transitioning to outcome rewards, accelerating convergence on effective reasoning paths. Experiments on seven benchmarks show consistent improvements over the state-of-the-art. Key advantages include: (1) Atom-Searcher scales computation at test-time. (2) Atomic Thought provides supervision anchors for RRMs, bridging deep research tasks and RRMs. (3) Atom-Searcher exhibits more interpretable, human-like reasoning patterns.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
