WebLINX：基于多轮对话的真实网站导航
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
AI总结：
本文提出对话式网页导航基准WebLINX，包含10万次交互，并设计检索式HTML剪枝模型，实验表明微调模型优于GPT-4V等零样本模型但泛化不足。
摘要：
我们提出了对话式网页导航问题，即数字代理控制网页浏览器并遵循用户指令，以多轮对话方式解决真实世界任务。为支持该问题，我们引入了WEBLINX——一个大规模基准，包含2300个专家演示中的10万次对话式网页导航交互。我们的基准覆盖了150多个真实网站上的广泛模式，可用于在不同场景下训练和评估代理。由于信息量巨大，大型语言模型（LLMs）无法实时处理整个网页。为解决这一瓶颈，我们设计了一个受检索启发的模型，通过对相关元素进行排序来高效剪枝HTML页面。我们使用选定的元素，结合截图和动作历史，评估多种模型在导航网页时复制人类行为的能力。我们的实验涵盖从小型纯文本模型到专有多模态LLMs。我们发现，较小的微调解码器超越了最佳的零样本LLMs（包括GPT-4V），也超越了显式在截图上预训练的较大微调多模态模型。然而，所有微调模型在泛化到未见过的网站时都遇到困难。我们的发现突显了对能够泛化到新环境的大型多模态模型的需求。我们的代码、数据和模型可供研究使用：https://mcgill-nlp.github.io/weblinx
英文摘要：
We propose the problem of conversational web navigation, where a digital agent controls a web browser and follows user instructions to solve real-world tasks in a multi-turn dialogue fashion. To support this problem, we introduce WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert demonstrations of conversational web navigation. Our benchmark covers a broad range of patterns on over 150 real-world websites and can be used to train and evaluate agents in diverse scenarios. Due to the magnitude of information present, Large Language Models (LLMs) cannot process entire web pages in real-time. To solve this bottleneck, we design a retrieval-inspired model that efficiently prunes HTML pages by ranking relevant elements. We use the selected elements, along with screenshots and action history, to assess a variety of models for their ability to replicate human behavior when navigating the web. Our experiments span from small text-only to proprietary multimodal LLMs. We find that smaller finetuned decoders surpass the best zero-shot LLMs (including GPT-4V), but also larger finetuned multimodal models which were explicitly pretrained on screenshots. However, all finetuned models struggle to generalize to unseen websites. Our findings highlight the need for large multimodal models that can generalize to novel settings. Our code, data and models are available for research: https://mcgill-nlp.github.io/weblinx
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
