WebAgent-R1：通过端到端多轮强化学习训练Web智能体
WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning
AI总结：
提出端到端多轮强化学习框架WebAgent-R1，通过异步轨迹生成和二元奖励训练Web智能体，在WebArena-Lite上显著提升任务成功率，并揭示思考提示与测试时扩展的有效性。
摘要：
虽然强化学习（RL）在增强大型语言模型（LLMs）方面已展现出显著成功，但其主要聚焦于单轮任务，如解决数学问题。由于在动态Web界面中进行长时域决策的复杂性，训练用于多轮交互的有效Web智能体仍然具有挑战性。在本工作中，我们提出了WebAgent-R1，一个简单而有效的端到端多轮RL框架，用于训练Web智能体。它通过异步生成多样化轨迹，完全由取决于任务成败的二元奖励引导，直接从与Web环境的在线交互中学习。在WebArena-Lite基准上的实验证明了WebAgent-R1的有效性，将Qwen-2.5-3B的任务成功率从6.1%提升至33.9%，将Llama-3.1-8B从8.5%提升至44.8%，显著优于现有最先进方法和强大的专有模型（如OpenAI o3）。深入分析揭示了基于思考的提示策略以及通过增加交互进行测试时扩展对Web任务的有效性。我们进一步通过引入两个变体，即WebAgent-R1-Zero和WebAgent-R1-CoT，研究了不同的RL初始化策略，这些变体突显了预热训练阶段（即行为克隆）的重要性，并为在Web智能体中融入长思维链（CoT）推理提供了见解。
英文摘要：
While reinforcement learning (RL) has demonstrated remarkable success in enhancing large language models (LLMs), it has primarily focused on single-turn tasks such as solving math problems. Training effective web agents for multi-turn interactions remains challenging due to the complexity of long-horizon decision-making across dynamic web interfaces. In this work, we present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework for training web agents. It learns directly from online interactions with web environments by asynchronously generating diverse trajectories, entirely guided by binary rewards depending on task success. Experiments on the WebArena-Lite benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to 44.8%, significantly outperforming existing state-of-the-art methods and strong proprietary models such as OpenAI o3. In-depth analyses reveal the effectiveness of the thinking-based prompting strategy and test-time scaling through increased interactions for web tasks. We further investigate different RL initialization policies by introducing two variants, namely WebAgent-R1-Zero and WebAgent-R1-CoT, which highlight the importance of the warm-up training stage (i.e., behavior cloning) and provide insights on incorporating long chain-of-thought (CoT) reasoning in web agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
