EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR
Meta；KAUST；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Zhenyu Li, Sai Kumar Dwivedi, Filip Maric, Carlos Chacon, Nadine Bertsch, Filippo Arcadu, Tomas Hodan, Michael Ramamonjisoa, Peter Wonka, Amy Zhao, Robin Kips, … 展开作者
Zhenyu Li, Sai Kumar Dwivedi, Filip Maric, Carlos Chacon, Nadine Bertsch, Filippo Arcadu, Tomas Hodan, Michael Ramamonjisoa, Peter Wonka, Amy Zhao, Robin Kips, Cem Keskin, Anastasia Tkach, Chenhongyi Yang
Egocentric human motion estimation is essential for AR/VR experiences, yet remains challenging due to limited body coverage from the egocentric viewpoint, frequent occlusions, and scarce labeled data. We present EgoPoseFormer v2, a method that addresses these challenges through two key contributions: (1) a transformer-based model for temporally consistent and spatially grounded body pose estimation, and (2) an auto-labeling system that enables the use of large unlabeled datasets for training. Our model is fully differentiable, introduces identity-conditioned queries, multi-view spatial refinement, causal temporal attention, and supports both keypoints and parametric body representations under a constant compute budget. The auto-labeling system scales learning to tens of millions of unlabeled frames via uncertainty-aware semi-supervised training. The system follows a teacher-student schema to generate pseudo-labels and guide training with uncertainty distillation, enabling the model to generalize to different environments. On the EgoBody3M benchmark, with a 0.8 ms latency on GPU, our model outperforms two state-of-the-art methods by 12.2% and 19.4% in accuracy, and reduces temporal jitter by 22.2% and 51.7%. Furthermore, our auto-labeling system further improves the wrist MPJPE by 13.1%.
Tianjin University（天津大学）；University of South Carolina（南卡罗来纳大学）
With the development of Augmented Reality (AR), egocentric action recognition (EAR) plays important role in accurately understanding demands from the user. However, EAR is designed to help recognize human-machine interaction in single egocentric view, thus difficult to capture interactions between two face-to-face AR users. Paired egocentric interaction recognition (PEIR) is the task to collaboratively recognize the interactions between two persons with the videos in their corresponding views. Unfortunately, existing PEIR methods always directly use linear decision function to fuse the feature…
Noldus Information Technology（诺达斯信息技术公司）；Utrecht University（乌得勒支大学）
In this work we employ multitask learning to capitalize on the structure that exists in related supervised tasks to train complex neural networks. It allows training a network for multiple objectives in parallel, in order to improve performance on at least one of them by capitalizing on a shared representation that is developed to accommodate more information than it otherwise would for a single task. We employ this idea to tackle action recognition in egocentric videos by introducing additional supervised tasks. We consider learning the verbs and nouns from which action labels consist of and …
EgoMusic-driven Human Dance Motion Estimation with Skeleton Mamba
FPT Software AI Center（FPT软件AI中心）；The University of Western Australia（西澳大学）；TU Wien（维也纳技术大学）；Meta；University of Arkansas（阿肯色大学）；National University of Singapore（新加坡国立大学）；University of Liverpool（利物浦大学）
Understanding-Enhanced Model Collaboration for Long-Tailed Egocentric Mistake Detection
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；School of Computer Science and Tech., University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Information Engineering, CAS（信息工程研究所，中国科学院）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
University of Oxford（牛津大学）；NAVER LABS Europe（NAVER LABS 欧洲）
Given a raw video sequence taken from a freely-moving camera, we study the problem of decomposing the observed 3D scene into a static background and a dynamic foreground containing the objects that move in the video sequence. This task is reminiscent of the classic background subtraction problem, but is significantly harder because all parts of the scene, static and dynamic, generate a large apparent motion due to the camera large viewpoint change. In particular, we consider egocentric videos and further separate the dynamic component into objects and the actor that observes and moves them. We…
Georgia Institute of Technology（佐治亚理工学院）；ETH Zürich（苏黎世联邦理工学院）；Zhejiang University（浙江大学）
We introduce a novel task of reconstructing a time series of second-person 3D human body meshes from monocular egocentric videos. The unique viewpoint and rapid embodied camera motion of egocentric videos raise additional technical barriers for human body capture. To address those challenges, we propose a simple yet effective optimization-based approach that leverages 2D observations of the entire video sequence and human-scene interaction constraint to estimate second-person human poses, shapes, and global motion that are grounded on the 3D environment captured from the egocentric view. We co…
Egocentric 3D human pose estimation with a single fisheye camera has drawn a significant amount of attention recently. However, existing methods struggle with pose estimation from in-the-wild images, because they can only be trained on synthetic data due to the unavailability of large-scale in-the-wild egocentric datasets. Furthermore, these methods easily fail when the body parts are occluded by or interacting with the surrounding scene. To address the shortage of in-the-wild data, we collect a large-scale in-the-wild egocentric dataset called Egocentric Poses in the Wild (EgoPW). This datase…
Egocentric 3D human pose estimation using a single fisheye camera has become popular recently as it allows capturing a wide range of daily activities in unconstrained environments, which is difficult for traditional outside-in motion capture with external cameras. However, existing methods have several limitations. A prominent problem is that the estimated poses lie in the local coordinate system of the fisheye camera, rather than in the world coordinate system, which is restrictive for many applications. Furthermore, these methods suffer from limited accuracy and temporal instability due to a…
We introduce the novel problem of anticipating a time series of future hand masks from egocentric video. A key challenge is to model the stochasticity of future head motions, which globally impact the head-worn camera video analysis. To this end, we propose a novel deep generative model -- EgoGAN, which uses a 3D Fully Convolutional Network to learn a spatio-temporal video representation for pixel-wise visual anticipation, generates future head motion using Generative Adversarial Network (GAN), and then predicts the future hand masks based on the video representation and the generated future h…
EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videos
Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；University of Science and Technology of China（中国科学技术大学）；Imperial College London（伦敦帝国学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input
MPI Informatics（MPI信息学研究所）；Saarland Informatics Campus（萨尔兰信息学园区）；Google（谷歌公司）；University of Pennsylvania（宾夕法尼亚大学）；Saarbrücken Research Center for Visual Computing, Interaction and Artificial Intelligence（萨尔布吕肯视觉计算、交互与人工智能研究中心）
Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能教育部重点实验室，上海交通大学AI研究院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究所，东部技术研究所）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）；MoE Key Lab of AI, School of Computer Science, Shanghai Jiao Tong University（人工智能教育部重点实验室，上海交通大学计算机学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Lenovo（联想公司）
EgoTraj: Real-World Egocentric Human Trajectory Dataset for Multimodal Prediction
Department of Civil, Architectural, and Environmental Engineering, The University of Texas at Austin（土木、建筑与环境工程系，德克萨斯大学奥斯汀分校）；Meta Reality Labs（Meta现实实验室）；School of Architecture, The University of Texas at Austin（建筑学院，德克萨斯大学奥斯汀分校）
Color-based two-hand 3D pose estimation in the global coordinate system is essential in many applications. However, there are very few datasets dedicated to this task and no existing dataset supports estimation in a non-laboratory environment. This is largely attributed to the sophisticated data collection process required for 3D hand pose annotations, which also leads to difficulty in obtaining instances with the level of visual diversity needed for estimation in the wild. Progressing towards this goal, a large-scale dataset Ego2Hands was recently proposed to address the task of two-hand segm…
ETH Zürich（苏黎世联邦理工学院）；Microsoft（微软）；Meta Reality Labs Research（Meta现实实验室）
Understanding social interactions from egocentric views is crucial for many applications, ranging from assistive robotics to AR/VR. Key to reasoning about interactions is to understand the body pose and motion of the interaction partner from the egocentric view. However, research in this area is severely hindered by the lack of datasets. Existing datasets are limited in terms of either size, capture/annotation modalities, ground-truth quality, or interaction diversity. We fill this gap by proposing EgoBody, a novel large-scale dataset for human pose, shape and motion estimation from egocentric…
StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video
Mohamed bin Zayed University of Artificial Intelligence（莫扎德·本·泽德人工智能大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Imperial College London（伦敦帝国理工学院）
Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos
IRMV Lab, the Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China（IRMV实验室，自动化系，上海交通大学，上海200240，中国）；College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China（智能科学与技术学院，国防科技大学，长沙410073，中国）；Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China（电子工程系，上海交通大学，上海200240，中国）
