VideoWebArena：用视频理解网页任务评估长上下文多模态智能体
VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks
AI总结：
本文提出 VideoWebArena 基准，以 2,021 个基于视频教程的网页任务评估长上下文多模态智能体，揭示其在事实与技能保持上的显著不足。
摘要：
视频常被用于学习或提取完成任务所需的信息，其方式不同于仅凭文本和静态图像所能提供的方式。然而，许多现有智能体基准忽视了长上下文视频理解，而是聚焦于文本或静态图像输入。为弥合这一差距，我们提出 VideoWebArena（VideoWA），这是一个用于评估长上下文多模态智能体视频理解能力的基准。VideoWA 包含 2,021 个基于手工制作视频教程的网页智能体任务，内容总时长接近四小时。针对该基准，我们定义了一套基于长上下文视频的智能体任务分类体系，包含两个主要关注领域：技能保持和事实保持。技能保持任务评估智能体能否利用给定的人类演示高效完成任务，而事实保持任务评估智能体能否从视频中检索与指令相关的信息以完成任务。我们发现，最佳模型在事实保持任务上取得 13.3% 的成功率，在事实保持问答对上取得 45.8% 的成功率，远低于人类分别达到的 73.9% 和 79.3%。在技能保持任务上，长上下文模型在有教程时的表现比无教程时更差，在 WebArena 任务中表现下降 5%，在 VisualWebArena 任务中下降 10.3%。我们的工作强调了提升长上下文多模态模型智能体能力的必要性，并为未来长上下文视频智能体的发展提供了测试平台。
英文摘要：
Videos are often used to learn or extract the necessary information to complete tasks in ways different than what text and static imagery alone can provide. However, many existing agent benchmarks neglect long-context video understanding, instead focusing on text or static image inputs. To bridge this gap, we introduce VideoWebArena (VideoWA), a benchmark for evaluating the capabilities of long-context multimodal agents for video understanding. VideoWA consists of 2,021 web agent tasks based on manually crafted video tutorials, which total almost four hours of content. For our benchmark, we define a taxonomy of long-context video-based agent tasks with two main areas of focus: skill retention and factual retention. While skill retention tasks evaluate whether an agent can use a given human demonstration to complete a task efficiently, the factual retention task evaluates whether an agent can retrieve instruction-relevant information from a video to complete a task. We find that the best model achieves 13.3% success on factual retention tasks and 45.8% on factual retention QA pairs, far below human performance at 73.9% and 79.3%, respectively. On skill retention tasks, long-context models perform worse with tutorials than without, exhibiting a 5% performance decrease in WebArena tasks and a 10.3% decrease in VisualWebArena tasks. Our work highlights the need to improve the agentic abilities of long-context multimodal models and provides a testbed for future development with long-context video agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
