Visual grounding associates textual descriptions with objects in an image. Conventional methods target third-person image inputs and named object queries. In applications such as AI assistants, the perspective shifts -- inputs are egocentric, and objects may be referred to implicitly through needs and intentions. To bridge this gap, we introduce EgoIntention, the first dataset for egocentric visual intention grounding. EgoIntention challenges multimodal LLMs to 1) understand and ignore unintended contextual objects and 2) reason about uncommon object functionalities. Benchmark results show that current models misidentify context objects and lack affordance understanding in egocentric views. We also propose Reason-to-Ground (RoG) instruction tuning; it enables hybrid training with normal descriptions and egocentric intentions with a chained intention reasoning and object grounding mechanism. RoG significantly outperforms naive finetuning and hybrid training on EgoIntention, while maintaining or slightly improving naive description grounding. This advancement enables unified visual grounding for egocentric and exocentric visual inputs while handling explicit object queries and implicit human intentions.
VS-Assistant: Versatile Surgery Assistant on the Demand of Surgeons
Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS（香港科学院人工智能与机器人中心（CAIR））；The Chinese University of Hong Kong（香港中文大学）；King’s College London（伦敦国王学院）
In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
State Key Laboratory of VR Technology and Systems, School of CSE, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学计算机科学与工程学院）；College of AI, Tsinghua University（清华大学人工智能学院）
Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges
Department of EEE（电子工程系）；The Hong Kong Polytechnic University（香港理工大学）；RayNeo；Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）；Nanyang Technological University（南洋理工大学）
Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions
Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）；University of Science and Technology Beijing (USTB)（北京科技大学）
A long-standing goal of intelligent assistants such as AR glasses/robots has been to assist users in affordance-centric real-world scenarios, such as "how can I run the microwave for 1 minute?". However, there is still no clear task definition and suitable benchmarks. In this paper, we define a new task called Affordance-centric Question-driven Task Completion, where the AI assistant should learn from instructional videos to provide step-by-step help in the user's view. To support the task, we constructed AssistQ, a new dataset comprising 531 question-answer samples from 100 newly filmed instr…
CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video
University College London（伦敦大学学院）；Imperial College London（帝国理工学院）；Durham University（杜伦大学）；Shanghai Jiao Tong University（上海交通大学）；Queen Mary University of London（伦敦玛丽女王大学）
Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time
Beijing Jiaotong University（北京交通大学）；Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences (CASIA)（基础模型研究中心、自动化研究所、中国科学院（CASIA））；Tencent Robotics X（腾讯机器人X）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology (HKUST)（计算机科学与工程系、香港科学与技术大学（HKUST））；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳学院）
EgoBlind: Towards Egocentric Visual Assistance for the Blind
National University of Singapore（新加坡国立大学）；Communication University of China（中国传媒大学）；University of Science and Technology of China（中国科学技术大学）；Hefei University of Technoloy（合肥工业大学）
Intention-Conditioned Long-Term Human Egocentric Action Forecasting
Technische Universität Wien (TU Wien)（维也纳技术大学）；Artificial Intelligence Graduate School (AIGS), Ulsan National Institute of Science and Technology (UNIST)（蔚山科学技术院人工智能研究生院）；Institute of Robotics and Mechatronics, German Aerospace Center（德国航空航天中心机器人与机电一体化研究所）
EgoSelf: From Memory to Personalized Egocentric Assistant
Eastern Institute of Technology（东部技术研究所）；Hong Kong Polytechnic University（香港理工大学）；Peking University（北京大学）；Commonwealth Scientific and Industrial Research Organization（澳大利亚联邦科学与工业研究组织）；University of Hong Kong（香港大学）
Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能教育部重点实验室，上海交通大学AI研究院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究所，东部技术研究所）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）；MoE Key Lab of AI, School of Computer Science, Shanghai Jiao Tong University（人工智能教育部重点实验室，上海交通大学计算机学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Lenovo（联想公司）
EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning
Tsinghua University（清华大学）；Tongji University（同济大学）；Renmin University of China（中国人民大学）；The University of Tokyo（东京大学）；Lenovo Group（联想集团）；Peking University（北京大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Shanghai Qi Zhi Institute（上海启智研究院）
APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants
Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology, Ningbo（宁波东方理工大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Zhongguancun Academy（中关村学院）；Dalian University of Technology（大连理工大学）；Beihang University（北京航空航天大学）；Hong Kong Polytechnic University（香港理工大学）
EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants
Fraunhofer IPK（弗劳恩霍夫研究所）；Technical University of Berlin（柏林技术大学）；University of Tübingen（图宾根大学）；RWTH Aachen University（亚琛工业大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
University of Pennsylvania（宾夕法尼亚大学）；Toyota Research Institute（丰田研究所）
Egocentric videos offer fine-grained information for high-fidelity modeling of human behaviors. Hands and interacting objects are one crucial aspect of understanding a viewer's behaviors and intentions. We provide a labeled dataset consisting of 11,243 egocentric images with per-pixel segmentation labels of hands and objects being interacted with during a diverse array of daily activities. Our dataset is the first to label detailed hand-object contact boundaries. We introduce a context-aware compositional data augmentation technique to adapt to out-of-distribution YouTube egocentric video. We …
Uni. of Bristol（布里斯托尔大学）；Uni. of Catania（卡塔尼亚大学）；Uni. of Toronto（多伦多大学）
First-person vision is gaining interest as it offers a unique viewpoint on people's interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict nonscripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North Ame…
MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos
IRMV Lab, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University and State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis, Shanghai Key Laboratory of Navigation and Location Based Services（IRMV实验室，自动化与智能感知学院，上海交通大学，航空集成与航空系统-of-Systems综合国家重点实验室，导航与定位服务重点实验室）；College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科技大学）；Department of Electronic Engineering, Shanghai Jiao Tong University（电子工程学院，上海交通大学）
Insight Center for Data Analytics（洞察数据分析中心）；Universitat Politecnica de Catalunya（加泰罗尼亚理工大学）
This work presents a retrieval pipeline and evaluation scheme for the problem of finding the last appearance of personal objects in a large dataset of images captured from a wearable camera. Each personal object is modelled by a small set of images that define a query for a visual search engine.The retrieved results are reranked considering the temporal timestamps of the images to increase the relevance of the later detections. Finally, a temporal interleaving of the results is introduced for robustness against false detections. The Mean Reciprocal Rank is proposed as a metric to evaluate this…
University of Catania（卡塔尼亚大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Although First Person Vision systems can sense the environment from the user's perspective, they are generally unable to predict his intentions and goals. Since human activities can be decomposed in terms of atomic actions and interactions with objects, intelligent wearable systems would benefit from the ability to anticipate user-object interactions. Even if this task is not trivial, the First Person Vision paradigm can provide important cues to address this challenge. We propose to exploit the dynamics of the scene to recognize next-active-objects before an object interaction begins. We trai…
