面向网页导航的双视图视觉上下文建模
Dual-View Visual Contextualization for Web Navigation
AI总结：
本文提出利用网页截图中HTML元素的文本与视觉双视图及其邻居元素进行上下文建模，以增强网页导航智能体的动作决策能力，并在Mind2Web数据集上验证了该方法在跨任务、跨网站和跨领域场景下均优于基线。
摘要：
自动网页导航旨在构建一个能够遵循语言指令、在真实网站上执行复杂多样任务的网页智能体。现有工作主要以HTML文档作为输入，这些文档定义了网页的内容和动作空间（即可操作元素与操作）。然而，HTML文档可能无法为每个元素提供清晰的任务相关上下文，这使得选择正确的（动作序列）变得困难。在本文中，我们提出通过网页截图中HTML元素的“双视图”来为其构建上下文：每个HTML元素在截图中都有其对应的边界框和视觉内容。我们基于以下洞察——网页开发者倾向于将任务相关的元素安排在网页上彼此邻近的位置以提升用户体验——提出利用文本和视觉特征，通过其邻居元素来为每个元素构建上下文。由此得到的HTML元素表示对智能体采取动作更具信息量。我们在最近发布的Mind2Web数据集上验证了我们的方法，该数据集涵盖了真实网站上多样化的导航领域和任务。我们的方法在所有场景中均一致优于基线，包括跨任务、跨网站和跨领域场景。
英文摘要：
Automatic web navigation aims to build a web agent that can follow language instructions to execute complex and diverse tasks on real-world websites. Existing work primarily takes HTML documents as input, which define the contents and action spaces (i.e., actionable elements and operations) of webpages. Nevertheless, HTML documents may not provide a clear task-related context for each element, making it hard to select the right (sequence of) actions. In this paper, we propose to contextualize HTML elements through their "dual views" in webpage screenshots: each HTML element has its corresponding bounding box and visual content in the screenshot. We build upon the insight -- web developers tend to arrange task-related elements nearby on webpages to enhance user experiences -- and propose to contextualize each element with its neighbor elements, using both textual and visual features. The resulting representations of HTML elements are more informative for the agent to take action. We validate our method on the recently released Mind2Web dataset, which features diverse navigation domains and tasks on real-world websites. Our method consistently outperforms the baseline in all the scenarios, including cross-task, cross-website, and cross-domain ones.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
