WatchAct: 行为引导的机器人操作基准
WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation
AI总结：
提出WatchAct基准，通过人类行为视频与指令配对，评估机器人对观察行为的推理能力，涵盖事件解析、程序推理、意图推断和情景记忆四个认知维度，实验显示现有系统性能远低于人类。
摘要：
与人类协作的机器人必须推理人类做了什么、顺序如何以及意图是什么。视频包含了语言未明确指定的空间布局、物体历史和手势，但当今的操作基准仅将指令与单张当前图像配对，无法评估对观察到的行为的推理。我们提出WatchAct，一个基于观察到的行为进行机器人操作的基准。每个实例将真实世界的人类动作视频和语言指令与对齐的模拟器场景和可执行的LIBERO任务配对，实现可扩展和可重复的评估。WatchAct包含3000个长时域实例，涵盖14个任务，分布在四个能力领域，这些领域源自观察其他智能体的认知需求：事件解析（事件定位）、程序结构恢复（程序推理）、推断未陈述的意图（隐式意图推理）以及跟踪场景如何被改变（情景推理）。我们进一步提出一种解耦评估协议，分别测量(i)视觉语言模型的视频到计划推理、(ii)在oracle计划下的策略执行，以及(iii)集成规划器-策略管道的完整任务完成。在仿真和Franka Research 3机器人上，当前系统远未解决WatchAct。最佳管道Gemini-3.1-Pro与$\pi_{0.5}$在仿真中仅达到16.3%的成功率（SR），在真实机器人上为14.0%。Gemini-3.1-Pro仅达到36.8%的计划SR（人类为97.1%），而$\pi_{0.5}$在oracle计划下仅达到21.5%的任务SR，在域外场景下降至10.6%。数据集和代码见此https URL。
英文摘要：
A robot working alongside people must reason about what they have done, in what order, and with what intent. Video carries the spatial layouts, object histories, and gestures that language leaves underspecified, yet today's manipulation benchmarks pair an instruction with a single current image, offering no way to evaluate reasoning over observed human behavior. We introduce WatchAct, a benchmark for robot manipulation grounded in observed human behavior. Each instance pairs a real-world human-action video and a language instruction with an aligned simulator scene and an executable LIBERO task, enabling scalable and reproducible evaluation. WatchAct comprises 3,000 long-horizon instances across 14 tasks in four capability domains drawn from the cognitive demands of watching another agent: parsing events (Event Grounding), recovering procedural structure (Procedural Reasoning), inferring unstated intent (Implicit Intent Inference), and tracking how the scene was changed (Episodic Reasoning). We further propose a disentangled evaluation protocol that separately measures (i)~video-to-plan reasoning by vision-language models, (ii)~policy execution under oracle plans, and (iii)~full task completion by integrated planner--policy pipelines. In both simulation and on a Franka Research 3 robot, current systems remain far from solving WatchAct. The best pipeline, Gemini-3.1-Pro with $π_{0.5}$, reaches only 16.3% Success Rate (SR) in simulation and 14.0% on the real robot. Gemini-3.1-Pro attains just 36.8% Plan SR (vs. 97.1% for humans), while $π_{0.5}$ reaches only 21.5% Task SR under oracle plans and drops to 10.6% on out-of-domain scenarios. Dataset and code are available at https://baiqi-li.github.io/watchact_page/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
