arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.01927cs.AIcs.MA
收藏

从基础到规划:网络代理中的瓶颈基准测试

From Grounding to Planning: Benchmarking Bottlenecks in Web Agents

Segev Shlomov, Ben wiesel, Aviad Sela, Ido Levy, Liane Galanti, Roy Abitbol

AI总结:

本文通过分析网络代理的规划和基础组件,发现规划是性能退化的主要原因,提出新的基准测试以改进代理能力。

摘要:

基于网页的通用代理正在越来越重要,用于与复杂的网页环境交互,但它们在现实世界网页应用中的性能仍然很差,即使使用最先进的前沿模型,也产生极低的准确性。我们发现这些代理可以分解为两个主要组件:规划和基础。然而,大多数现有研究将这些代理视为黑箱,专注于端到端评估,这阻碍了有意义的改进。我们加强了规划和基础组件之间的区别,并通过在Mind2Web数据集上精炼实验进行新的分析。我们的工作为每个组件分别提出一个新的基准,识别限制代理性能的瓶颈和痛点。与普遍假设相反,我们的发现表明基础不是主要瓶颈,可以使用当前技术有效解决。相反,主要挑战在于规划组件,这是性能退化的主要来源。通过这一分析,我们提供了新的见解,并展示了改进网络代理能力的实用建议,为更可靠的代理铺平了道路。

英文摘要:

General web-based agents are increasingly essential for interacting with complex web environments, yet their performance in real-world web applications remains poor, yielding extremely low accuracy even with state-of-the-art frontier models. We observe that these agents can be decomposed into two primary components: Planning and Grounding. Yet, most existing research treats these agents as black boxes, focusing on end-to-end evaluations which hinder meaningful improvements. We sharpen the distinction between the planning and grounding components and conduct a novel analysis by refining experiments on the Mind2Web dataset. Our work proposes a new benchmark for each of the components separately, identifying the bottlenecks and pain points that limit agent performance. Contrary to prevalent assumptions, our findings suggest that grounding is not a significant bottleneck and can be effectively addressed with current techniques. Instead, the primary challenge lies in the planning component, which is the main source of performance degradation. Through this analysis, we offer new insights and demonstrate practical suggestions for improving the capabilities of web agents, paving the way for more reliable agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑