Egocentric interactive world models are essential for augmented reality and embodied AI, where visual generation must respond to user input with low latency, geometric consistency, and long-term stability. We study egocentric interaction generation from a single scene image under free-space hand gestures, aiming to synthesize photorealistic videos in which hands enter the scene, interact with objects, and induce plausible world dynamics under head motion. This setting introduces fundamental challenges, including distribution shift between free-space gestures and contact-heavy training data, ambiguity between hand motion and camera motion in monocular views, and the need for arbitrary-length video generation. We present Hand2World, a unified autoregressive framework that addresses these challenges through occlusion-invariant hand conditioning based on projected 3D hand meshes, allowing visibility and occlusion to be inferred from scene context rather than encoded in the control signal. To stabilize egocentric viewpoint changes, we inject explicit camera geometry via per-pixel Plücker-ray embeddings, disentangling camera motion from hand motion and preventing background drift. We further develop a fully automated monocular annotation pipeline and distill a bidirectional diffusion model into a causal generator, enabling arbitrary-length synthesis. Experiments on three egocentric interaction benchmarks show substantial improvements in perceptual quality and 3D consistency while supporting camera control and long-horizon interactive generation.
Korea University（高丽大学）；Drexel University（德雷塞尔大学）；DMLab
In this paper, a deep learning-based model for 3D human motion generation from the text is proposed via gesture action classification and an autoregressive model. The model focuses on generating special gestures that express human thinking, such as waving and nodding. To achieve the goal, the proposed method predicts expression from the sentences using a text classification model based on a pretrained language model and generates gestures using the gate recurrent unit-based autoregressive model. Especially, we proposed the loss for the embedding space for restoring raw motions and generating i…
Mobile virtual reality (VR) head mounted displays (HMD) have become popular among consumers in recent years. In this work, we demonstrate real-time egocentric hand gesture detection and localization on mobile HMDs. Our main contributions are: 1) A novel mixed-reality data collection tool to automatic annotate bounding boxes and gesture labels; 2) The largest-to-date egocentric hand gesture and bounding box dataset with more than 400,000 annotated frames; 3) A neural network that runs real time on modern mobile CPUs, and achieves higher than 76% precision on gesture recognition across 8 classes…
OMRON SINIC X（欧姆龙 SINIC X 研究院）；Kyoto University（京都大学）
We introduce an on-ground Pedestrian World Model, a computational model that can predict how pedestrians move around an observer in the crowd on the ground plane, but from just the egocentric-views of the observer. Our model, InCrowdFormer, fully leverages the Transformer architecture by modeling pedestrian interaction and egocentric to top-down view transformation with attention, and autoregressively predicts on-ground positions of a variable number of people with an encoder-decoder architecture. We encode the uncertainties arising from unknown pedestrian heights with latent codes to predict …
Natural interaction with virtual objects in AR/VR environments makes for a smooth user experience. Gestures are a natural extension from real world to augmented space to achieve these interactions. Finding discriminating spatio-temporal features relevant to gestures and hands in ego-view is the primary challenge for recognising egocentric gestures. In this work we propose a data driven end-to-end deep learning approach to address the problem of egocentric gesture recognition, which combines an ego-hand encoder network to find ego-hand features, and a recurrent neural network to discern tempora…
Recognizing people by faces and other biometrics has been extensively studied in computer vision. But these techniques do not work for identifying the wearer of an egocentric (first-person) camera because that person rarely (if ever) appears in their own first-person view. But while one's own face is not frequently visible, their hands are: in fact, hands are among the most common objects in one's own field of view. It is thus natural to ask whether the appearance and motion patterns of people's hands are distinctive enough to recognize them. In this paper, we systematically study the possibil…
University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；Princeton University（普林斯顿大学）；Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）
Head-mounted device-based human-computer interaction often requires egocentric recognition of hand gestures and fingertips detection. In this paper, a unified approach of egocentric hand gesture recognition and fingertip detection is introduced. The proposed algorithm uses a single convolutional neural network to predict the probabilities of finger class and positions of fingertips in one forward propagation. Instead of directly regressing the positions of fingertips from the fully connected layer, the ensemble of the position of fingertips is regressed from the fully convolutional network. Su…
Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos
IRMV Lab, the Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China（IRMV实验室，自动化系，上海交通大学，上海200240，中国）；College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China（智能科学与技术学院，国防科技大学，长沙410073，中国）；Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China（电子工程系，上海交通大学，上海200240，中国）
HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Stanford University（斯坦福大学）；Nanyang Technological University（南洋理工大学）；Meta Reality Labs Research（Meta现实实验室）
mmEgoHand: Egocentric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMU
School of Software Engineering, Xi’an Jiaotong University（软件工程学院，西安交通大学）；School of Cyber Science and Engineering, Xi’an Jiaotong University（网络科学与工程学院，西安交通大学）；School of Computer Science and Technology, Xi’an Jiaotong University（计算机科学与技术学院，西安交通大学）；College of Computer Science and Technology, Zhejiang University（计算机科学与技术学院，浙江大学）
Carnegie Mellon University（卡内基梅隆大学）；Michigan State University（密歇根州立大学）
Accurately identifying hands in images is a key sub-task for human activity understanding with wearable first-person point-of-view cameras. Traditional hand segmentation approaches rely on a large corpus of manually labeled data to generate robust hand detectors. However, these approaches still face challenges as the appearance of the hand varies greatly across users, tasks, environments or illumination conditions. A key observation in the case of many wearable applications and interfaces is that, it is only necessary to accurately detect the user's hands in a specific situational context. Bas…
Macau University of Science and Technology（澳门科技大学）；Communication University of Zhejiang（浙江传媒学院）；Ningbo University of Finance & Economics（宁波财经学院）
Speech-driven gesture synthesis is a field of growing interest in virtual human creation. However, a critical challenge is the inherent intricate one-to-many mapping between speech and gestures. Previous studies have explored and achieved significant progress with generative models. Notwithstanding, most synthetic gestures are still vastly less natural. This paper presents DiffMotion, a novel speech-driven gesture synthesis architecture based on diffusion models. The model comprises an autoregressive temporal encoder and a denoising diffusion probability Module. The encoder extracts the tempor…
GestureFAR: Streaming Co-Speech Gesture Generation with Flow Autoregression
University of Rochester（罗切斯特大学）；University of Tokyo（东京大学）；University of California Santa Cruz（加州大学圣克鲁兹分校）；University of California Los Angeles（加州大学洛杉矶分校）；Meta
Trinity College Dublin（都柏林圣三一学院）；Intel Labs China（英特尔中国研究院）
Egocentric gestures are the most natural form of communication for humans to interact with wearable devices such as VR/AR helmets and glasses. A major issue in such scenarios for real-world applications is that may easily become necessary to add new gestures to the system e.g., a proper VR system should allow users to customize gestures incrementally. Traditional deep learning methods require storing all previous class samples in the system and training the model again from scratch by incorporating previous samples and new samples, which costs humongous memory and significantly increases compu…
Universitat Jaume I（海梅一世大学）；Institute of New Imaging Technologies（新型成像技术研究所）
Non-verbal communication plays a particularly important role in a wide range of scenarios in Human-Robot Interaction (HRI). Accordingly, this work addresses the problem of human gesture recognition. In particular, we focus on head and eye gestures, and adopt an egocentric (first-person) perspective using eyewear cameras. We argue that this egocentric view may offer a number of conceptual and technical benefits over scene- or robot-centric perspectives. A motion-based recognition approach is proposed, which operates at two temporal granularities. Locally, frame-to-frame homographies are estimat…
ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation
Department of Computer Science, University of Texas at Dallas, Richardson, TX, USA.（德克萨斯大学达拉斯分校计算机科学系）；College of Computing, Georgia Institute of Technology, Atlanta, GA, USA（佐治亚理工学院计算机学院）；Department of Computer Science, University of Wisconsin-Madison, Madison, WI, USA（威斯康星大学麦迪逊分校计算机科学系）；Allen School of Computer Science, University of Washington, USA（华盛顿大学阿伦计算机科学学院）
EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras
National Research Center for Optical Instrumentation, Zhejiang University（浙江大学光学仪器国家研究中心）；School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学人工智能与机器人学院和机器人视觉感知与控制技术国家工程研究中心）；Ant Group Company Ltd.（蚂蚁集团有限公司）
EgoEvGesture: Gesture Recognition Based on Egocentric Event Camera
State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University（浙江大学极端光效与仪器国家重点实验室）；School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学机器人视觉感知与控制技术国家工程研究中心）
Macau University of Science and Technology（澳门科技大学）；Communication University of Zhejiang（浙江传媒学院）；Zhejiang Lab（之江实验室）；Communication University of China（中国传媒大学）；Zhejiang University（浙江大学）；Zhejiang Institute of Economics and Trade（浙江经贸职业技术学院）
MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantization
Communication University of China（中国传媒大学）；China Mobile Research Institute（中国移动研究院）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；The Chinese University of Hong Kong（香港中文大学）
University of Chinese Academy of Sciences（中国科学院大学）；City University of Hong Kong（香港城市大学）；South China University of Technology（华南理工大学）
Egocentric videos, which mainly record the activities carried out by the users of the wearable cameras, have drawn much research attentions in recent years. Due to its lengthy content, a large number of ego-related applications have been developed to abstract the captured videos. As the users are accustomed to interacting with the target objects using their own hands while their hands usually appear within their visual fields during the interaction, an egocentric hand detection step is involved in tasks like gesture recognition, action recognition and social interaction understanding. In this …
GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院（EPFL））；University of Bern（伯尔尼大学）；Swiss Data Science Center（瑞士数据科学中心）；University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）
Egocentric videos are characterised by their ability to have the first person view. With the popularity of Google Glass and GoPro, use of egocentric videos is on the rise. Recognizing action of the wearer from egocentric videos is an important problem. Unstructured movement of the camera due to natural head motion of the wearer causes sharp changes in the visual field of the egocentric camera causing many standard third person action recognition techniques to perform poorly on such videos. Objects present in the scene and hand gestures of the wearer are the most important cues for first person…
VarGes: Improving Variation in Co-Speech 3D Gesture Generation via StyleCLIPS
Communication University of China（中国传媒大学）；Samsung Research America（三星美国研究院）；Hainan International College, Communication University of China（中国传媒大学海南国际学院）；Beihang University（北京航空航天大学）；Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）
Analysis of the hands in egocentric vision: A survey
KITE – Toronto Rehab – University Health Network（KITE – 多伦多康复 – 大学健康网络）；Institute of Biomaterials and Biomedical Engineering (IBBME), University of Toronto（多伦多大学生物材料与生物医学工程研究所）；Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto（多伦多大学爱德华·S·罗杰斯 Sr. 电气与计算机工程系）；Rehabilitation Sciences Institute, University of Toronto（多伦多大学康复科学研究所）
Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time
Beijing Jiaotong University（北京交通大学）；Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences (CASIA)（基础模型研究中心、自动化研究所、中国科学院（CASIA））；Tencent Robotics X（腾讯机器人X）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology (HKUST)（计算机科学与工程系、香港科学与技术大学（HKUST））；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳学院）
Institute for Creative Technologies, University of Southern California（南加州大学创意技术研究所）；Electronics and Telecommunications Research Institute（韩国电子通信研究院）
Synthesizing realistic co-speech gestures is an important and yet unsolved problem for creating believable motions that can drive a humanoid robot to interact and communicate with human users. Such capability will improve the impressions of the robots by human users and will find applications in education, training, and medical services. One challenge in learning the co-speech gesture model is that there may be multiple viable gesture motions for the same speech utterance. The deterministic regression methods can not resolve the conflicting samples and may produce over-smoothed or damped motio…
EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
Shanghai Jiao Tong University（上海交通大学）；Alibaba Group（阿里巴巴集团）；Tianji KernalMind Co., Ltd.（天机芯智有限公司）；The Hong Kong University of Science and Technology（香港科技大学）；Southeast University（东南大学）；Renmin University of China（中国人民大学）；The University of Tokyo（东京大学）
BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset
Salesforce Research（Salesforce 研究院）；University of Maryland（马里兰大学）；Virginia Tech（弗吉尼亚理工大学）；New York University（纽约大学）；University of Washington（华盛顿大学）；UC Davis（加州大学戴维斯分校）
The high prevalence of work-related musculoskeletal disorders (WMSDs) could be addressed by optimizing Human-Robot Collaboration (HRC) frameworks for manufacturing applications. In this context, this paper proposes two hypotheses for ergonomically effective task delegation and HRC. The first hypothesis states that it is possible to quantify ergonomically professional tasks using motion data from a reduced set of sensors. Then, the most dangerous tasks can be delegated to a collaborative robot. The second hypothesis is that by including gesture recognition and spatial adaptation, the ergonomics…
