IntentRL: 通过强化学习训练主动的用户意图代理以进行开放性深度研究
IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learning
AI总结：
IntentRL通过强化学习训练主动用户意图代理，提升开放性深度研究的意图识别与任务性能。
摘要：
深度研究（DR）代理通过自主检索和合成来自大规模网络语料库的证据，将大型语言模型（LLMs）扩展到非参数知识之外，从而生成长篇报告，实现了长周期的代理范式。然而，与实时对话助手不同，DR计算成本高且耗时长，造成自主性与交互的困境：在模糊用户查询上高自主性往往导致执行时间延长且结果不满意。为此，我们提出了IntentRL框架，该框架训练主动代理在开始长周期研究前澄清潜在的用户意图。为克服开放性研究数据稀缺的问题，我们引入了一个可扩展的流程，通过浅层到深层意图细化图将少量种子样本扩展为高质量对话轮次。我们进一步采用两阶段强化学习（RL）策略：第一阶段在离线对话上应用RL以高效学习一般用户交互行为，而第二阶段使用训练好的代理和用户模拟器进行在线回放以增强对多样化用户反馈的适应能力。广泛实验表明，IntentRL显著提高了意图命中率和下游任务性能，优于封闭源DR代理内置的澄清模块和主动LLM基线。
英文摘要：
Deep Research (DR) agents extend Large Language Models (LLMs) beyond parametric knowledge by autonomously retrieving and synthesizing evidence from large web corpora into long-form reports, enabling a long-horizon agentic paradigm. However, unlike real-time conversational assistants, DR is computationally expensive and time-consuming, creating an autonomy-interaction dilemma: high autonomy on ambiguous user queries often leads to prolonged execution with unsatisfactory outcomes. To address this, we propose IntentRL, a framework that trains proactive agents to clarify latent user intents before starting long-horizon research. To overcome the scarcity of open-ended research data, we introduce a scalable pipeline that expands a few seed samples into high-quality dialogue turns via a shallow-to-deep intent refinement graph. We further adopt a two-stage reinforcement learning (RL) strategy: Stage I applies RL on offline dialogues to efficiently learn general user-interaction behavior, while Stage II uses the trained agent and a user simulator for online rollouts to strengthen adaptation to diverse user feedback. Extensive experiments show that IntentRL significantly improves both intent hit rate and downstream task performance, outperforming the built-in clarify modules of closed-source DR agents and proactive LLM baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
