Analyzing hand-object interaction in egocentric vision facilitates VR/AR applications and human-robot policy transfer. Existing research has mostly focused on modeling the behavior paradigm of interactive actions (i.e., ``how to interact''). However, the more challenging and fine-grained problem of capturing the critical moments of contact and separation between the hand and the target object (i.e., ``when to interact'') is still underexplored, which is crucial for immersive interactive experiences in mixed reality and robotic motion planning. Therefore, we formulate this problem as temporal interaction localization (TIL). Some recent works extract semantic masks as TIL references, but suffer from inaccurate object grounding and cluttered scenarios. Although current temporal action localization (TAL) methods perform well in detecting verb-noun action segments, they rely on category annotations during training and exhibit limited precision in localizing hand-object contact/separation moments. To address these issues, we propose a novel zero-shot approach dubbed EgoLoc to localize hand-object contact and separation timestamps in egocentric videos. EgoLoc introduces hand-dynamics-guided sampling to generate high-quality visual prompts. It exploits the vision-language model to identify contact/separation attributes, localize specific timestamps, and provide closed-loop feedback for further refinement. EgoLoc eliminates the need for object masks and verb-noun taxonomies, leading to generalizable zero-shot implementation. Comprehensive experiments on the public dataset and our novel benchmarks demonstrate that EgoLoc achieves plausible TIL for egocentric videos. It is also validated to effectively facilitate multiple downstream applications in egocentric vision and robotic manipulation tasks. Code and relevant data will be released at https://github.com/IRMVLab/EgoLoc.
Indiana University（印第安纳大学）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）
We analyze egocentric views of attended objects from infants. This paper shows 1) empirical evidence that children's egocentric views have more diverse distributions compared to adults' views, 2) we can computationally simulate the infants' distribution, and 3) the distribution is beneficial for training more generalized image classifiers not only for infant egocentric vision but for third-person computer vision.
Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Qi Zhi Institute（上海期智研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Institute of Technology（北京理工大学）
GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院（EPFL））；University of Bern（伯尔尼大学）；Swiss Data Science Center（瑞士数据科学中心）；University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）
Stanford University（斯坦福大学）；NVIDIA Research（英伟达研究院）；The University of Hong Kong（香港大学）
The ability to forecast human-environment collisions from egocentric observations is vital to enable collision avoidance in applications such as VR, AR, and wearable assistive robotics. In this work, we introduce the challenging problem of predicting collisions in diverse environments from multi-view egocentric videos captured from body-mounted cameras. Solving this problem requires a generalizable perception system that can classify which human body joints will collide and estimate a collision region heatmap to localize collisions in the environment. To achieve this, we propose a transformer-…
Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
Zhejiang University（浙江大学）；Beijing Zhongguancun Academy（北京中关村学院）；University of Science and Technology of China（中国科学技术大学）；Eastern Institute of Technology（东方理工学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
The University of Hong Kong（香港大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；Knowin AI（考拉悠然科技）
