超越十轮：利用大规模异步强化学习解锁长程智能体搜索
Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL
AI总结：
本文提出ASearcher，通过大规模异步强化学习训练搜索智能体，突破轮次限制实现长程搜索，在xBench和GAIA上超越现有开源32B模型。
摘要：
近期基于大语言模型的智能体通过整合外部工具，在处理复杂的知识密集型任务方面展现出卓越能力。在众多工具选择中，搜索工具在获取海量外部知识方面发挥着关键作用。然而，开源智能体在实现专家级搜索智能方面仍存在不足，即解决模糊查询、生成精确搜索、分析结果并进行彻底探索的能力。现有方法在可扩展性、效率和数据质量方面存在不足。例如，现有在线强化学习方法中的小轮次限制（如≤10）限制了复杂策略的学习。本文介绍了ASearcher，一个用于搜索智能体大规模强化学习训练的开源项目。我们的主要贡献包括：（1）可扩展的全异步强化学习训练，能够在保持高训练效率的同时实现长程搜索。（2）一个基于提示的LLM智能体，能够自主合成高质量且具有挑战性的问答对，从而创建大规模问答数据集。通过强化学习训练，我们基于提示的QwQ-32B智能体取得了显著改进，在xBench和GAIA上分别获得78.0%和34.3%的Avg@4提升。值得注意的是，我们的智能体展现出极长程搜索能力，训练期间工具调用超过100轮，输出token超过400k。凭借简单的智能体设计且无需外部LLM，ASearcher-Web-QwQ在xBench上达到51.1的Avg@4分数，在GAIA上达到58.7，超越了现有的开源32B智能体。最后，我们还展示了ASearcher-Web-QwQ能够以零样本迁移方式和测试时搜索达到使用外部摘要工具的商业系统的性能。我们在https://github.com/inclusionAI/ASearcher开源了我们的模型、训练数据和代码。
英文摘要：
Recent advancements in LLM-based agents have demonstrated remarkable capabilities in handling complex, knowledge-intensive tasks by integrating external tools. Among diverse choices of tools, search tools play a pivotal role in accessing vast external knowledge. However, open-source agents still fall short of achieving expert-level Search Intelligence, the ability to resolve ambiguous queries, generate precise searches, analyze results, and conduct thorough exploration. Existing approaches fall short in scalability, efficiency, and data quality. For example, small turn limits in existing online RL methods, e.g. <=10, restrict complex strategy learning. This paper introduces ASearcher, an open-source project for large-scale RL training of search agents. Our key contributions include: (1) Scalable fully asynchronous RL training that enables long-horizon search while maintaining high training efficiency. (2) A prompt-based LLM agent that autonomously synthesizes high-quality and challenging QAs, creating a large-scale QA dataset. Through RL training, our prompt-based QwQ-32B agent achieves substantial improvements, with 78.0% and 34.3% Avg@4 gains on xBench and GAIA, respectively. Notably, our agent exhibits extreme long-horizon search, with tool calls exceeding 100 turns and output tokens exceeding 400k during training time. With a simple agent design and no external LLMs, ASearcher-Web-QwQ achieves Avg@4 scores of 51.1 on xBench and 58.7 on GAIA, surpassing existing open-source 32B agents. Finally, we also show that ASearcher-Web-QwQ could achieve performance of commercial systems using external summary tool in a zero-shot transfer manner and test-time search. We open-source our models, training data, and codes in https://github.com/inclusionAI/ASearcher.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
