arXiv 2312.158202023-12-25cs.CV

WebVLN：网站上的视觉与语言导航

WebVLN: Vision-and-Language Navigation on Websites

Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）

Qi Chen, Dileepa Pitawela, Chongyang Zhao, Gengze Zhou, Hsiang-Ting Chen, Qi Wu

备注：: Accepted by AAAI2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出WebVLN任务，利用基于问题的指令训练代理在网站上导航，并引入WebVLN-Net方法结合HTML信息，数据集WebVLN-v1验证其优于现有方法。

摘要：

视觉与语言导航（VLN）任务旨在使AI代理能够准确理解并遵循自然语言指令，在真实环境中导航，最终到达特定目标位置。我们认识到一个有望将VLN扩展到类似导航任务的机会，该任务在我们的日常生活中具有重要意义，尽管是在虚拟领域：在互联网上导航网站。本文提出了一项名为“网站上的视觉与语言导航”（WebVLN）的新任务，我们使用基于问题的指令来训练代理，模拟用户自然浏览网站的方式。与仅关注视觉和指令（语言）的现有VLN任务不同，WebVLN代理进一步考虑网页特有的底层内容，如HTML，这些内容在渲染的网页上不可见，但包含丰富的视觉和文本信息。为此，我们贡献了一个数据集WebVLN-v1，并引入了一种名为“网站感知VLN网络”（WebVLN-Net）的新方法，该方法建立在最先进的VLN技术基础上。实验结果表明，WebVLN-Net优于当前的VLN和网页相关导航方法。我们相信，新WebVLN任务及其数据集的引入将在VLN领域建立一个新的维度，并为更广泛的视觉与语言研究社区做出贡献。代码可在https://github.com/WebVLN/WebVLN获取。

英文摘要：

Vision-and-Language Navigation (VLN) task aims to enable AI agents to accurately understand and follow natural language instructions to navigate through real-world environments, ultimately reaching specific target locations. We recognise a promising opportunity to extend VLN to a comparable navigation task that holds substantial significance in our daily lives, albeit within the virtual realm: navigating websites on the Internet. This paper proposes a new task named Vision-and-Language Navigation on Websites (WebVLN), where we use question-based instructions to train an agent, emulating how users naturally browse websites. Unlike the existing VLN task that only pays attention to vision and instruction (language), the WebVLN agent further considers underlying web-specific content like HTML, which could not be seen on the rendered web pages yet contains rich visual and textual information. Toward this goal, we contribute a dataset, WebVLN-v1, and introduce a novel approach called Website-aware VLN Network (WebVLN-Net), which is built upon the foundation of state-of-the-art VLN techniques. Experimental results show that WebVLN-Net outperforms current VLN and web-related navigation methods. We believe that the introduction of the new WebVLN task and its dataset will establish a new dimension within the VLN domain and contribute to the broader vision-and-language research community. The code is available at: https://github.com/WebVLN/WebVLN.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。