For efficient human-agent interaction, an agent should proactively recognize their target user and prepare for upcoming interactions. We formulate this challenging problem as the novel task of jointly forecasting a person's intent to interact with the agent, their attitude towards the agent and the action they will perform, from the agent's (egocentric) perspective. So we propose \emph{SocialEgoNet} - a graph-based spatiotemporal framework that exploits task dependencies through a hierarchical multitask learning approach. SocialEgoNet uses whole-body skeletons (keypoints from face, hands and body) extracted from only 1 second of video input for high inference speed. For evaluation, we augment an existing egocentric human-agent interaction dataset with new class labels and bounding box annotations. Extensive experiments on this augmented dataset, named JPL-Social, demonstrate \emph{real-time} inference and superior performance (average accuracy across all tasks: 83.15\%) of our model outperforming several competitive baselines. The additional annotations and code will be available upon acceptance.
In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
State Key Laboratory of VR Technology and Systems, School of CSE, Beihang University（虚拟现实技术与系统国家重点实验室，北京航空航天大学计算机科学与工程学院）；College of AI, Tsinghua University（清华大学人工智能学院）
Intention-Conditioned Long-Term Human Egocentric Action Forecasting
Technische Universität Wien (TU Wien)（维也纳技术大学）；Artificial Intelligence Graduate School (AIGS), Ulsan National Institute of Science and Technology (UNIST)（蔚山科学技术院人工智能研究生院）；Institute of Robotics and Mechatronics, German Aerospace Center（德国航空航天中心机器人与机电一体化研究所）
EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos
IRMV Lab, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University and State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis, Shanghai Key Laboratory of Navigation and Location Based Services（IRMV实验室，自动化与智能感知学院，上海交通大学，航空集成与航空系统-of-Systems综合国家重点实验室，导航与定位服务重点实验室）；College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科技大学）；Department of Electronic Engineering, Shanghai Jiao Tong University（电子工程学院，上海交通大学）
EgoPriMo: Egocentric Motion Generation for Interactive Humanoid Control
Tianjin University（天津大学）；Zhongguancun Academy（中关村学院）；Beihang University（北京航空航天大学）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；DeepCybo
Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI
Shenzhen International Graduate School Tsinghua University Shenzhen China（深圳国际研究生院清华大学深圳中国）；Independent Researcher London United Kingdom（独立研究者伦敦英国）；Queen Mary University of London London United Kingdom（女王玛丽大学伦敦英国）；Imperial College London London United Kingdom（帝国理工学院伦敦英国）；University Of Surrey Guildford United Kingdom（Surrey大学Guildford英国）
How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Prediction
Khoury College of Computer Sciences, Northeastern University, Boston（北德文斯克学院，东北大学，波士顿）；Korea Advanced Institute of Science and Technology, Daejeon（韩国科学技术院，大田）；Department of Mathematics and Computer Science, University of Catania, Italy（卡塔尼亚大学数学与计算机科学系，意大利）
University of Catania（卡塔尼亚大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Although First Person Vision systems can sense the environment from the user's perspective, they are generally unable to predict his intentions and goals. Since human activities can be decomposed in terms of atomic actions and interactions with objects, intelligent wearable systems would benefit from the ability to anticipate user-object interactions. Even if this task is not trivial, the First Person Vision paradigm can provide important cues to address this challenge. We propose to exploit the dynamics of the scene to recognize next-active-objects before an object interaction begins. We trai…
Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Peking University（北京大学）；Tsinghua University（清华大学）
Understanding human tasks through video observations is an essential capability of intelligent agents. The challenges of such capability lie in the difficulty of generating a detailed understanding of situated actions, their effects on object states (i.e., state changes), and their causal dependencies. These challenges are further aggravated by the natural parallelism from multi-tasking and partial observations in multi-agent collaboration. Most prior works leverage action localization or future prediction as an indirect metric for evaluating such task understanding from videos. To make a dire…
Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
HKU Musketeers Foundation Institute of Data Science, The University of Hong Kong（香港大学穆斯克特基金会数据科学研究所，香港大学）；Department of Electrical and Electronic Engineering, The University of Hong Kong（电气电子工程系，香港大学）
Honda Research Institute USA（本田美国研究所）；University of California, Berkeley（加州大学伯克利分校）；Honda R&D Co., Ltd.（本田技术研究所）
Recent advances in trajectory prediction have shown that explicit reasoning about agents' intent is important to accurately forecast their motion. However, the current research activities are not directly applicable to intelligent and safety critical systems. This is mainly because very few public datasets are available, and they only consider pedestrian-specific intents for a short temporal horizon from a restricted egocentric view. To this end, we propose LOKI (LOng term and Key Intentions), a novel large-scale dataset that is designed to tackle joint trajectory and intention prediction for …
Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions
Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）；University of Science and Technology Beijing (USTB)（北京科技大学）
Georgia Institute of Technology（佐治亚理工学院）；ETH Zürich（苏黎世联邦理工学院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
We address the challenging task of anticipating human-object interaction in first person videos. Most existing methods ignore how the camera wearer interacts with the objects, or simply consider body motion as a separate modality. In contrast, we observe that the international hand movement reveals critical information about the future activity. Motivated by this, we adopt intentional hand movement as a future representation and propose a novel deep network that jointly models and predicts the egocentric hand motion, interaction hotspots and future action. Specifically, we consider the future …
School of Intelligence Science and Technology, Peking University（北京理工大学智能科学与技术学院）；National Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；AIR, Tsinghua University（清华大学人工智能研究院）
