VisualWebBench：多模态大语言模型在网页理解与定位方面进化到了何种程度？
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
AI总结：
本文提出多模态网页理解与定位基准VisualWebBench，包含7类任务和1.5K个真实网页实例，系统评估14个开源MLLMs及主流闭源模型，揭示了当前模型在文本丰富环境定位和低分辨率输入上的不足。
摘要：
多模态大语言模型（MLLMs）在与网页相关的任务中已展现出潜力，但由于缺乏全面的基准，评估它们在网页领域的表现仍然是一个挑战。现有基准要么是为通用多模态任务设计的，无法捕捉网页的独特特征，要么专注于端到端的网页智能体任务，无法衡量诸如OCR、理解和定位等细粒度能力。在本文中，我们引入了VisualWebBench，这是一个多模态基准，旨在评估MLLMs在各种网页任务上的能力。VisualWebBench由七个任务组成，包含来自139个真实网站的1.5K个人工整理的实例，覆盖87个子领域。我们在VisualWebBench上评估了14个开源MLLMs、Gemini Pro、Claude-3系列和GPT-4V(ision)，揭示了显著的挑战和性能差距。进一步的分析突出了当前MLLMs的局限性，包括在文本丰富的环境中定位能力不足，以及在低分辨率图像输入下表现欠佳。我们相信VisualWebBench将为研究社区提供宝贵的资源，并有助于为网页相关应用创建更强大、更多功能的MLLMs。
英文摘要：
Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks. Existing benchmarks are either designed for general multimodal tasks, failing to capture the unique characteristics of web pages, or focus on end-to-end web agent tasks, unable to measure fine-grained abilities such as OCR, understanding, and grounding. In this paper, we introduce \bench{}, a multimodal benchmark designed to assess the capabilities of MLLMs across a variety of web tasks. \bench{} consists of seven tasks, and comprises 1.5K human-curated instances from 139 real websites, covering 87 sub-domains. We evaluate 14 open-source MLLMs, Gemini Pro, Claude-3 series, and GPT-4V(ision) on \bench{}, revealing significant challenges and performance gaps. Further analysis highlights the limitations of current MLLMs, including inadequate grounding in text-rich environments and subpar performance with low-resolution image inputs. We believe \bench{} will serve as a valuable resource for the research community and contribute to the creation of more powerful and versatile MLLMs for web-related applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
