HoloAssist：面向真实世界交互式 AI 助手的第一视角人类交互数据集
HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
AI总结：
本文提出大规模第一视角双人交互数据集 HoloAssist，包含 166 小时多模态数据与动作、对话标注，并建立错误检测、介入预测和手部预测基准。
摘要：
构建一个能够在真实世界中感知、推理并与人类协作的交互式 AI 助手，长期以来一直是 AI 领域的追求。本工作属于一项更广泛研究努力的一部分，该努力旨在开发能够交互式引导人类在物理世界中完成任务的智能体。作为朝此方向迈出的第一步，我们提出 HoloAssist，这是一个大规模第一视角人类交互数据集，其中两人协作完成物理操作任务。任务执行者佩戴一台混合现实头显执行任务，该设备采集七路同步数据流。任务指导者实时观看执行者的第一视角视频，并通过语言指导他们。通过为数据补充动作和对话标注，并观察不同参与者丰富的行为，我们提出了关于人类指导者如何纠正错误、介入任务完成过程，以及如何将其指令锚定到环境中的关键洞见。HoloAssist 涵盖由 350 个不同的指导者—执行者配对采集的 166 小时数据。此外，我们构建并给出了错误检测、介入类型预测和手部预测方面的基准，并进行了详细分析。我们期望 HoloAssist 将为构建能够在真实世界中与人类流畅协作的 AI 助手提供重要资源。数据可在 https://holoassist.github.io/ 下载。
英文摘要：
Building an interactive AI assistant that can perceive, reason, and collaborate with humans in the real world has been a long-standing pursuit in the AI community. This work is part of a broader research effort to develop intelligent agents that can interactively guide humans through performing tasks in the physical world. As a first step in this direction, we introduce HoloAssist, a large-scale egocentric human interaction dataset, where two people collaboratively complete physical manipulation tasks. The task performer executes the task while wearing a mixed-reality headset that captures seven synchronized data streams. The task instructor watches the performer's egocentric video in real time and guides them verbally. By augmenting the data with action and conversational annotations and observing the rich behaviors of various participants, we present key insights into how human assistants correct mistakes, intervene in the task completion procedure, and ground their instructions to the environment. HoloAssist spans 166 hours of data captured by 350 unique instructor-performer pairs. Furthermore, we construct and present benchmarks on mistake detection, intervention type prediction, and hand forecasting, along with detailed analysis. We expect HoloAssist will provide an important resource for building AI assistants that can fluidly collaborate with humans in the real world. Data can be downloaded at https://holoassist.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
