A robot's ability to anticipate the 3D action target location of a hand's movement from egocentric videos can greatly improve safety and efficiency in human-robot interaction (HRI). While previous research predominantly focused on semantic action classification or 2D target region prediction, we argue that predicting the action target's 3D coordinate could pave the way for more versatile downstream robotics tasks, especially given the increasing prevalence of headset devices. This study expands EgoPAT3D, the sole dataset dedicated to egocentric 3D action target prediction. We augment both its size and diversity, enhancing its potential for generalization. Moreover, we substantially enhance the baseline algorithm by introducing a large pre-trained model and human prior knowledge. Remarkably, our novel algorithm can now achieve superior prediction outcomes using solely RGB images, eliminating the previous need for 3D point clouds and IMU input. Furthermore, we deploy our enhanced baseline algorithm on a real-world robotic platform to illustrate its practical utility in straightforward HRI tasks. The demonstrations showcase the real-world applicability of our advancements and may inspire more HRI use cases involving egocentric vision. All code and data are open-sourced and can be found on the project website.
Face versus Body Tracking for Human-Robot Interaction: An Egocentric Dataset
Furhat Robotics；University of Naples Federico II（那不勒斯费德里科二世大学）；Division of Speech, Music and Hearing, KTH Royal Institute of Technology（语音、音乐和听觉研究所，皇家理工学院）
ZeroWBC: Learning Natural Whole-Body Humanoid Interaction from Human Egocentric Data
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；Northwestern Polytechnical University（西北工业大学）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；TeleAI, China Telecom（TeleAI，中国电信）
EgoGuide: Egocentric Guidance for Efficient Robot-Free Demonstration Collection and Learning
Shanghai Jiao Tong University（上海交通大学）；Shanghai Innovation Institute（上海创新研究院）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）
EgoAVFlow: Robot Policy Learning with Active Vision from Human Egocentric Videos via 3D Flow
School of Interactive Computing, Georgia Institute of Technology（交互计算学院，佐治亚理工学院）；InnoCORE AI-Transformed Aerospace Research Center, KAIST（AI转型航空航天研究中心，韩国成均馆大学）
Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data
AIM3 Lab, Renmin University of China（中国人民大学AIM3实验室）；Qwen Team, Alibaba Inc.（阿里巴巴通义千问团队）；ShanghaiTech University（上海科技大学）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）；Beijing University of Aeronautics and Astronautics（北京航空航天大学）
PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；Harbin Institute of Technology（哈尔滨工业大学）；Huazhong University of Science and Technology（华中科技大学）
EgoMoD: Predicting Global Maps of Dynamics from Local Egocentric Observations
University of Turku, Finland（芬兰图尔库大学）；Centre for Artificial Intelligence, Zürich University of Applied Sciences, Winterthur, Switzerland（瑞士应用科学大学人工智能中心）；Instituto de Investigación en Ingeniería de Aragón, Universidad de Zaragoza, Spain（西班牙阿拉贡工程研究所，萨拉戈萨大学）
Co-training with Ego-centric Video and Demonstration for Robot Navigation Task
Department of Informatics, Graduate School of Informatics, Kyoto University（信息学系，京都大学研究生院）；Spatial Robotics Research Center, Fujitsu Limited（空间机器人研究中心，富士通有限公司）
TPT-Bench: A Large-Scale, Long-Term and Robot-Egocentric Dataset for Benchmarking Target Person Tracking
Robotics and Computer Vision Laboratory, Southern University of Science and Technology, Shenzhen, China（机器人与计算机视觉实验室，南方科技大学，深圳，中国）；Human-Robot Interfaces and Interaction Laboratory, Italian Institute of Technology, Geona, Italy（人机交互实验室，意大利理工学院，Geona，意大利）；Robotics Perception and Intelligence Laboratory, Southern University of Science and Technology, Shenzhen, China（机器人感知与智能实验室，南方科技大学，深圳，中国）；Chen Kar-Shun Robotics Institute, Hong Kong University of Science and Technology, Hong Kong SAR, China（陈卡尔顺机器人研究所，香港科技大学，香港特别行政区，中国）
DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Egocentric Whole-Body Human Demonstrations
The Hong Kong University of Science and Technology（香港科技大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；Beihang University（北京航空航天大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Developing Vision-Language-Action Model from Egocentric Videos
Kyoto University（京都大学）；National Institute of Informatics（国家信息研究所）；Institute of Science Tokyo（东京科学研究所）；NII LLMC（日本信息机构语言模型中心）；Sony Interactive Entertainment（索尼互动娱乐）
