Reality Fusion: Robust Real-time Immersive Mobile Robot Teleoperation with Volumetric Visual Data Fusion
accelerator division at the Deutsche Elektronen Synchrotron DESY（德意志电子同步辐射研究中心加速器部门）；Human-Computer Interaction group at Hamburg University（汉堡大学人机交互小组）
Ke Li, Reinhard Bacher, Susanne Schmidt, Wim Leemans, Frank Steinicke
We introduce Reality Fusion, a novel robot teleoperation system that localizes, streams, projects, and merges a typical onboard depth sensor with a photorealistic, high resolution, high framerate, and wide field of view (FoV) rendering of the complex remote environment represented as 3D Gaussian splats (3DGS). Our framework enables robust egocentric and exocentric robot teleoperation in immersive VR, with the 3DGS effectively extending spatial information of a depth sensor with limited FoV and balancing the trade-off between data streaming costs and data visual quality. We evaluated our framework through a user study with 24 participants, which revealed that Reality Fusion leads to significantly better user performance, situation awareness, and user preferences. To support further research and development, we provide an open-source implementation with an easy-to-replicate custom-made telepresence robot, a high-performance virtual reality 3DGS renderer, and an immersive robot control package. (Source code: https://github.com/uhhhci/RealityFusion)
University of Tsukuba（筑波大学）；Universite de Picardie Jules Verne（皮卡第儒勒·凡尔纳大学）；National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）
In immersive humanoid robot teleoperation, there are three main shortcomings that can alter the transparency of the visual feedback: the lag between the motion of the operator's and robot's head due to network communication delays or slow robot joint motion. This latency could cause a noticeable delay in the visual feedback, which jeopardizes the embodiment quality, can cause dizziness, and affects the interactivity resulting in operator frequent motion pauses for the visual feedback to settle; (ii) the mismatch between the camera's and the headset's field-of-views (FOV), the former having gen…
CaFe-TeleVision: A Coarse-to-Fine Teleoperation System with Immersive Situated Visualization for Enhanced Ergonomics
Department of Mechanical and Automation Engineering, T-Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong SAR（机械与自动化工程系、T-Stone机器人研究所、香港中文大学（深圳））
Institute of Computer Science II – Computer Graphics, University of Bonn（波恩大学计算机科学II系——计算机图形学）；Institute of Computer Science VI – Autonomous Intelligent Systems, University of Bonn（波恩大学计算机科学VI系——自主智能系统）
Applications like disaster management and industrial inspection often require experts to enter contaminated places. To circumvent the need for physical presence, it is desirable to generate a fully immersive individual live teleoperation experience. However, standard video-based approaches suffer from a limited degree of immersion and situation awareness due to the restriction to the camera view, which impacts the navigation. In this paper, we present a novel VR-based practical system for immersive robot teleoperation and scene exploration. While being operated through the scene, a robot captu…
Microsoft Mixed Reality and AI Lab（微软混合现实与人工智能实验室）；ETH Zürich（苏黎世联邦理工学院）；University of Haifa（海法大学）；NVIDIA（英伟达）
Spatial computing -- the ability of devices to be aware of their surroundings and to represent this digitally -- offers novel capabilities in human-robot interaction. In particular, the combination of spatial computing and egocentric sensing on mixed reality devices enables them to capture and understand human actions and translate these to actions with spatial meaning, which offers exciting new possibilities for collaboration between humans and robots. This paper presents several human-robot systems that utilize these capabilities to enable novel robot use cases: mission planning for inspecti…
Robotic avatars promise immersive teleoperation with human-like manipulation and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic bimanual arm configuration with dexterous hands. The remote human operator drives the arms and fingers through an exoskeleton-based operator station, which provides force feedback both at the wrist and for each finger. The robot torso is mounted on a holonomic base, providing locomotion capability in typ…
Imperial College London（伦敦帝国理工学院）；Indonesian Institute of Sciences - LIPI（印度尼西亚科学院）
In this work, we propose a novel mobile rescue robot equipped with an immersive stereoscopic teleperception and a teleoperation control. This robot is designed with the capability to perform safely a casualty-extraction procedure. We have built a proof-of-concept mobile rescue robot called ResQbot for the experimental platform. An approach called "loco-manipulation" is used to perform the casualty-extraction procedure using the platform. The performance of this robot is evaluated in terms of task accomplishment and safety by conducting a mock rescue experiment. We use a custom-made human-sized…
Imperial College London（帝国理工学院）；Dyson School of Design Engineering（戴森设计工程学院）
Achieving successful robotic manipulation is an essential step towards robots being widely used in industry and home settings. Recently, many learning-based methods have been proposed to tackle this challenge, with imitation learning showing great promise. However, imperfect demonstrations and a lack of feedback from teleoperation systems may lead to poor or even unsafe results. In this work we explore the effect of demonstrator force feedback on imitation learning, using a feedback glove and a robot arm to render fingertip-level and palm-level forces, respectively. 10 participants recorded 5 …
VR-DAgger: Immersive VR for Dexterous Data Collection and Uncertainty-Guided On-Policy Correction
Robotics Systems Lab（机器人系统实验室）；ETH Zürich（苏黎世联邦理工学院）；ETH AI Center（ETH人工智能中心）；ETH Augmented Reality Research Lab（ETH增强现实研究实验室）；ETH Mobility Initiative（ETH移动性倡议）；ANYbotics AG（ANYbotics公司）；Swiss Federal Railways（瑞士联邦铁路）
Human-Exoskeleton Kinematic Calibration to Improve Hand Tracking for Dexterous Teleoperation
Walker Department of Mechanical Engineering, The University of Texas at Austin（德克萨斯大学机械工程系）；Sony Group Corporation, Tokyo, Japan（索尼集团公司，日本东京）；Meta Reality Labs Research, Redmond, WA, USA（Meta现实实验室研究）
A fundamental challenge in teaching robots is to provide an effective interface for human teachers to demonstrate useful skills to a robot. This challenge is exacerbated in dexterous manipulation, where teaching high-dimensional, contact-rich behaviors often require esoteric teleoperation tools. In this work, we present Holo-Dex, a framework for dexterous manipulation that places a teacher in an immersive mixed reality through commodity VR headsets. The high-fidelity hand pose estimator onboard the headset is used to teleoperate the robot and collect demonstrations for a variety of general-pur…
The Role of Embodiment in Intuitive Whole-Body Teleoperation for Mobile Manipulation
Computer Science Department, TU Darmstadt（图恩-达姆施塔特大学计算机科学系）；University of São Paulo（圣保罗大学）；Centre for Cognitive Science, TU Darmstadt（图恩-达姆施塔特大学认知科学中心）；Systems AI for Robot Learning, German Research Center for AI (DFKI)（德国人工智能研究中心（DFKI）机器人学习系统AI）
Robots that solve complex tasks in environments too dangerous for humans to enter are desperately needed, e.g. for search and rescue applications. We describe our mobile manipulation robot Momaro, with which we participated successfully in the DARPA Robotics Challenge. It features a unique locomotion design with four legs ending in steerable wheels, which allows it both to drive omnidirectionally and to step over obstacles or climb. Furthermore, we present advanced communication and teleoperation approaches, which include immersive 3D visualization, and 6D tracking of operator head and arm mot…
Dexterous manipulation with anthropomorphic robot hands remains a challenging problem in robotics because of the high-dimensional state and action spaces and complex contacts. Nevertheless, skillful closed-loop manipulation is required to enable humanoid robots to operate in unstructured real-world environments. Reinforcement learning (RL) has traditionally imposed enormous interaction data requirements for optimizing such complex control problems. We introduce a new framework that leverages recent advances in GPU-based simulation along with the strength of imitation learning in guiding policy…
