It is especially challenging to achieve real-time human motion tracking on a standalone VR Head-Mounted Display (HMD) such as Meta Quest and PICO. In this paper, we propose HMD-Poser, the first unified approach to recover full-body motions using scalable sparse observations from HMD and body-worn IMUs. In particular, it can support a variety of input scenarios, such as HMD, HMD+2IMUs, HMD+3IMUs, etc. The scalability of inputs may accommodate users' choices for both high tracking accuracy and easy-to-wear. A lightweight temporal-spatial feature learning network is proposed in HMD-Poser to guarantee that the model runs in real-time on HMDs. Furthermore, HMD-Poser presents online body shape estimation to improve the position accuracy of body joints. Extensive experimental results on the challenging AMASS dataset show that HMD-Poser achieves new state-of-the-art results in both accuracy and real-time performance. We also build a new free-dancing motion dataset to evaluate HMD-Poser's on-device performance and investigate the performance gap between synthetic data and real-captured sensor data. Finally, we demonstrate our HMD-Poser with a real-time Avatar-driving application on a commercial HMD. Our code and free-dancing motion dataset are available https://pico-ai-team.github.io/hmd-poser
SSD-Poser: Avatar Pose Estimation with State Space Duality from Sparse Observations
Academy for Engineering & Technology, Fudan University（复旦大学工程与技术学院）；College of Vocational and Technical Teacher Education, Shanghai Polytechnic University（上海理工大学职业技术与教师教育学院）
University of Science and Technology of China（中国科学技术大学）；Carnegie Mellon University（卡内基梅隆大学）
In recent years, head-mounted near-eye display devices have become the key hardware foundation for virtual reality and augmented reality. Thus head-mounted gaze tracking technology has received attention as an essential part of human-computer interaction. However, unavoidable slippage of head-mounted devices (HMD) often results higher gaze tracking errors and hinders the practical usage of HMD. To tackle this problem, we propose a slippage-robust gaze tracking for near-eye display method based on the aspheric eyeball model and accurately compute the eyeball optical axis and rotation center. We…
Fraunhofer Institute for Computer Graphics Research IGD（弗劳恩霍夫计算机图形研究所（IGD））；TU Darmstadt（达姆施塔特工业大学）；The Norwegian Colour and Visual Computing Laboratory, NTNU（挪威色彩与视觉计算实验室，挪威科技大学（NTNU））；Norwegian Biometrics Laboratory, NTNU（挪威生物识别实验室，挪威科技大学（NTNU））；Mathematical and Applied Visual Computing（数学与应用视觉计算（TU Darmstadt））；Mathematical and Applied Visual Computing, TU Darmstadt（数学与应用视觉计算，达姆施塔特工业大学）
Augmented and virtual reality is being deployed in different fields of applications. Such applications might involve accessing or processing critical and sensitive information, which requires strict and continuous access control. Given that Head-Mounted Displays (HMD) developed for such applications commonly contains internal cameras for gaze tracking purposes, we evaluate the suitability of such setup for verifying the users through iris recognition. In this work, we first evaluate a set of iris recognition algorithms suitable for HMD devices by investigating three well-established handcrafte…
Progressive Inertial Poser: Progressive Real-Time Kinematic Chain Estimation for 3D Full-Body Pose from Three IMU Sensors
School of Communication Engineering, Hangzhou Dianzi University（杭州电子科技大学通信工程学院）；Key Laboratory of Micro-nano Sensing and IoT of Wenzhou, Wenzhou Institute of Hangzhou Dianzi University（杭州电子科技大学温州市微纳传感与物联网重点实验室）；School of Control Science and Engineering, Tiangong University（天津工业大学控制科学与工程学院）；College of Business, Lishui University（丽水学院商学院）；Costar Intelligent Optoelectronics Technology Co.,Ltd（Costar智能光电技术有限公司）；School of Automation, Hangzhou Dianzi University（杭州电子科技大学自动化学院）；Faculty of Applied Sciences, Macao Polytechnic University（澳门理工学院应用科学学院）；School of software and BNRist, Tsinghua University（清华大学软件学院）
Reality Labs Research, Meta（Meta现实实验室研究部）；Meta AI Research（Meta人工智能研究院）
Real-time tracking of human body motion is crucial for interactive and immersive experiences in AR/VR. However, very limited sensor data about the body is available from standalone wearable devices such as HMDs (Head Mounted Devices) or AR glasses. In this work, we present a reinforcement learning framework that takes in sparse signals from an HMD and two controllers, and simulates plausible and physically valid full body motions. Using high quality full body motion as dense supervision during training, a simple policy network can learn to output appropriate torques for the character to balanc…
Technical University Munich（慕尼黑工业大学）；Stanford University（斯坦福大学）；University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Max-Planck-Institute for Informatics（马克斯·普朗克信息学研究所）
We propose FaceVR, a novel image-based method that enables video teleconferencing in VR based on self-reenactment. State-of-the-art face tracking methods in the VR context are focused on the animation of rigged 3d avatars. While they achieve good tracking performance the results look cartoonish and not real. In contrast to these model-based approaches, FaceVR enables VR teleconferencing using an image-based technique that results in nearly photo-realistic outputs. The key component of FaceVR is a robust algorithm to perform real-time facial motion capture of an actor who is wearing a head-moun…
A head-mounted display (HMD) is a portable and interactive display device. With the development of 5G technology, it may become a general-purpose computing platform in the future. Human-computer interaction (HCI) technology for HMDs has also been of significant interest in recent years. In addition to tracking gestures and speech, tracking human eyes as a means of interaction is highly effective. In this paper, we propose two UnityEyes-based convolutional neural network models, UEGazeNet and UEGazeNet*, which can be used for input images with low resolution and high resolution, respectively. T…
Xerox Research Centre Europe（施乐欧洲研究中心）；Inria（法国国家信息与自动化研究所）
We introduce an approach for spatio-temporal human action localization using sparse spatial supervision. Our method leverages the large amount of annotated humans available today and extracts human tubes by combining a state-of-the-art human detector with a tracking-by-detection approach. Given these high-quality human tubes and temporal supervision, we select positive and negative tubes with very sparse spatial supervision, i.e., only one spatially annotated frame per instance. The selected tubes allow us to effectively learn a spatio-temporal action detector based on dense trajectories or CN…
