We propose a method for incorporating object interaction and human body dynamics into the task of 3D ego-pose estimation using a head-mounted camera. We use a kinematics model of the human body to represent the entire range of human motion, and a dynamics model of the body to interact with objects inside a physics simulator. By bringing together object modeling, kinematics modeling, and dynamics modeling in a reinforcement learning (RL) framework, we enable object-aware 3D ego-pose estimation. We devise several representational innovations through the design of the state and action space to incorporate 3D scene context and improve pose estimation quality. We also construct a fine-tuning step to correct the drift and refine the estimated human-object interaction. This is the first work to estimate a physically valid 3D full-body interaction sequence with objects (e.g., chairs, boxes, obstacles) from egocentric videos. Experiments with both controlled and in-the-wild settings show that our method can successfully extract an object-conditioned 3D ego-pose sequence that is consistent with the laws of physics.
INTACT: Ego-Guided Typed Sparse Evidence Retrieval for Heterogeneous Collaborative Perception
National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（多谱信息智能处理国家级重点实验室，人工智能与自动化学院，华中科技大学）；Zhejiang Normal University（浙江师范大学）
We propose a method for object-aware 3D egocentric pose estimation that tightly integrates kinematics modeling, dynamics modeling, and scene object information. Unlike prior kinematics or dynamics-based approaches where the two components are used disjointly, we synergize the two approaches via dynamics-regulated training. At each timestep, a kinematic model is used to provide a target pose using video evidence and simulation state. Then, a prelearned dynamics model attempts to mimic the kinematic pose in a physics simulator. By comparing the pose instructed by the kinematic model against the …
EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR（香港理工大学电子与电气工程系）；Division of Emerging Interdisciplinary Areas (EMIA), The Hong Kong University of Science and Technology, Hong Kong SAR（香港理工大学新兴跨学科领域研究中心）
To drive safely in complex traffic environments, autonomous vehicles need to make an accurate prediction of the future trajectories of nearby heterogeneous traffic agents (i.e., vehicles, pedestrians, bicyclists, etc). Due to the interactive nature, human drivers are accustomed to infer what the future situations will become if they are going to execute different maneuvers. To fully exploit the impacts of interactions, this paper proposes a ego-planning guided multi-graph convolutional network (EPG-MGCN) to predict the trajectories of heterogeneous agents using both historical trajectory infor…
CoopDiff: A Diffusion-Guided Approach for Cooperation under Corruptions
School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）；School of Cybersecurity, Tianjin University（天津大学网络安全学院）；School of Future Technology, Tianjin University（天津大学未来技术学院）
Decoupling Ego-Motion from Target Dynamics via Dual-Interval Motion Cues for UAV Detection
Department of Robotics, School of Advanced Manufacturing and Robotics（机器人学院，先进制造与机器人学院）；State Key Laboratory of Turbulence and Complex Systems（湍流与复杂系统国家重点实验室）；Peking University（北京大学）；Great Bay University（大湾大学）
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室OpenGVLab）；The University of Tokyo（东京大学）；Nanjing University（南京大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；USTC（中国科学技术大学）；SIAT, CAS（中国科学院深圳先进技术研究院）
University of Zagreb, Faculty of Electrical Engineering and Computing（萨格勒布大学电气工程与计算学院）；Yamagata University, Faculty of Science（山形大学理学部）
This paper proposes to use keypoints as a self-supervision clue for learning depth map estimation from a collection of input images. As ground truth depth from real images is difficult to obtain, there are many unsupervised and self-supervised approaches to depth estimation that have been proposed. Most of these unsupervised approaches use depth map and ego-motion estimations to reproject the pixels from the current image into the adjacent image from the image collection. Depth and ego-motion estimations are evaluated based on pixel intensity differences between the correspondent original and …
Beijing Institute of Artificial Intelligence（北京人工智能研究院）；Beijing University of Technology（北京工业大学）
3D action recognition is referred to as the classification of action sequences which consist of 3D skeleton joints. While many research work are devoted to 3D action recognition, it mainly suffers from three problems: highly complicated articulation, a great amount of noise, and a low implementation efficiency. To tackle all these problems, we propose a real-time 3D action recognition framework by integrating the locally aggregated kinematic-guided skeletonlet (LAKS) with a supervised hashing-by-analysis (SHA) model. We first define the skeletonlet as a few combinations of joint offsets groupe…
Michigan State University（密歇根州立大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Perceiving the physical world in 3D is fundamental for self-driving applications. Although temporal motion is an invaluable resource to human vision for detection, tracking, and depth perception, such features have not been thoroughly utilized in modern 3D object detectors. In this work, we propose a novel method for monocular video-based 3D object detection which carefully leverages kinematic motion to improve precision of 3D localization. Specifically, we first propose a novel decomposition of object orientation as well as a self-balancing 3D confidence. We show that both components are crit…
Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis
Texas A&M University（德克萨斯A&M大学）；University of Minnesota（明尼苏达大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Texas at Austin（得克萨斯大学奥斯汀分校）；Amazon（亚马逊）；State University of New York at Stony Brook（纽约州立大学石溪分校）
EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videos
Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；University of Science and Technology of China（中国科学技术大学）；Imperial College London（伦敦帝国学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras
National Research Center for Optical Instrumentation, Zhejiang University（浙江大学光学仪器国家研究中心）；School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（湖南大学人工智能与机器人学院和机器人视觉感知与控制技术国家工程研究中心）；Ant Group Company Ltd.（蚂蚁集团有限公司）
