arXiv 2605.172622026-05-17cs.CV

EgoIntrospect: 一个用于用户中心内部状态推理的注视数据集和基准

EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning

Tsinghua University（清华大学）；Tongji University（同济大学）；Renmin University of China（中国人民大学）；The University of Tokyo（东京大学）；Lenovo Group（联想集团）；Peking University（北京大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Shanghai Qi Zhi Institute（上海启智研究院）

Zeyu Wang, Chang Liu, Eduardus Tjitrahardja, Yuntao Wang, Borislav Pavlov, Fangfei Gou, Jose Manuel Davila, Dai Shi, Ran Xu, Yue Pan, Jiayi Tan, Shuting Chang, … 展开作者

Zeyu Wang, Chang Liu, Eduardus Tjitrahardja, Yuntao Wang, Borislav Pavlov, Fangfei Gou, Jose Manuel Davila, Dai Shi, Ran Xu, Yue Pan, Jiayi Tan, Shuting Chang, Qi Wang, Jinzhao Li, Jiacheng Hua, Yifei Huang, Jingwei Sun, Yu Zhang, Liuxin Zhang, Guocai Yao, Jia Jia, Yin Li, Qianying Wang, Yuanchun Shi, Miao Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出EgoIntrospect数据集，用于研究用户中心内部状态推理，通过自注释揭示用户与AI助手的交互意图，评估多模态大语言模型在从注视观察中推理用户内部状态的能力。

摘要：

尽管在注视视频数据集和基准方面已有大量努力，但理解用户内部状态，这对于实现无缝的AI助手体验至关重要，仍被忽视。在本文中，我们介绍了EgoIntrospect，这是第一个在用户驱动场景中捕捉的注视数据集，具有自我注释，明确揭示用户与AI助手的交互意图。EgoIntrospect使用跨设备设置收集，提供了同步的视频、音频、注视、运动和生理信号。它包含60名受试者180小时的记录，平均每人记录3小时。利用EgoIntrospect，我们正式化了一套围绕用户内部状态的任务，包括情感体验、交互意图和认知记忆。我们进一步处理注释以构建评估现代多模态大语言模型从注视观察中推理用户内部状态能力的基准。在我们基准上的实验表明，现有的多模态大语言模型难以有效利用多模态信号来推断用户的主观内部状态。该数据集和注释将向公众开放，以促进注视视觉和可穿戴AI助手的研究。项目页面：https://ego-introspect.github.io/

英文摘要：

Despite extensive efforts on egocentric video datasets and benchmarks, understanding users' internal states, which is crucial for enabling seamless AI assistant experiences, remains largely overlooked. In this work, we introduce EgoIntrospect, the first egocentric dataset captured in user-driven scenarios with self-annotations that explicitly reveal users' interactive intentions with AI assistants. EgoIntrospect was collected using a cross-device setup, providing synchronized video, audio, gaze, motion, and physiological signals. It consists of 180 hours of recordings from 60 subjects, with an average recording duration of 3 hours per subject. Leveraging EgoIntrospect, we formalize a suite of tasks centered on user internal states, including affective experience, interactive intent, and cognitive memory. We further process the annotations to construct benchmarks that evaluate the ability of modern multimodal large language models to reason about users' internal states from egocentric observations. Experiments on our benchmark suggest that existing multimodal large language models struggle to effectively leverage multimodal signals to infer users' subjective internal states. The dataset and annotations will be made publicly available to advance research in egocentric vision and wearable AI assistants. Project page: https://ego-introspect.github.io/

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。