Teleoperation serves as a powerful method for collecting on-robot data essential for robot learning from demonstrations. The intuitiveness and ease of use of the teleoperation system are crucial for ensuring high-quality, diverse, and scalable data. To achieve this, we propose an immersive teleoperation system Open-TeleVision that allows operators to actively perceive the robot's surroundings in a stereoscopic manner. Additionally, the system mirrors the operator's arm and hand movements on the robot, creating an immersive experience as if the operator's mind is transmitted to a robot embodiment. We validate the effectiveness of our system by collecting data and training imitation learning policies on four long-horizon, precise tasks (Can Sorting, Can Insertion, Folding, and Unloading) for 2 different humanoid robots and deploy them in the real world. The system is open-sourced at: https://robot-tv.github.io/
CaFe-TeleVision: A Coarse-to-Fine Teleoperation System with Immersive Situated Visualization for Enhanced Ergonomics
Department of Mechanical and Automation Engineering, T-Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong SAR（机械与自动化工程系、T-Stone机器人研究所、香港中文大学（深圳））
Human-Exoskeleton Kinematic Calibration to Improve Hand Tracking for Dexterous Teleoperation
Walker Department of Mechanical Engineering, The University of Texas at Austin（德克萨斯大学机械工程系）；Sony Group Corporation, Tokyo, Japan（索尼集团公司，日本东京）；Meta Reality Labs Research, Redmond, WA, USA（Meta现实实验室研究）
Imperial College London（帝国理工学院）；Dyson School of Design Engineering（戴森设计工程学院）
Achieving successful robotic manipulation is an essential step towards robots being widely used in industry and home settings. Recently, many learning-based methods have been proposed to tackle this challenge, with imitation learning showing great promise. However, imperfect demonstrations and a lack of feedback from teleoperation systems may lead to poor or even unsafe results. In this work we explore the effect of demonstrator force feedback on imitation learning, using a feedback glove and a robot arm to render fingertip-level and palm-level forces, respectively. 10 participants recorded 5 …
Institute of Computer Science II – Computer Graphics, University of Bonn（波恩大学计算机科学II系——计算机图形学）；Institute of Computer Science VI – Autonomous Intelligent Systems, University of Bonn（波恩大学计算机科学VI系——自主智能系统）
Applications like disaster management and industrial inspection often require experts to enter contaminated places. To circumvent the need for physical presence, it is desirable to generate a fully immersive individual live teleoperation experience. However, standard video-based approaches suffer from a limited degree of immersion and situation awareness due to the restriction to the camera view, which impacts the navigation. In this paper, we present a novel VR-based practical system for immersive robot teleoperation and scene exploration. While being operated through the scene, a robot captu…
Reality Fusion: Robust Real-time Immersive Mobile Robot Teleoperation with Volumetric Visual Data Fusion
accelerator division at the Deutsche Elektronen Synchrotron DESY（德意志电子同步辐射研究中心加速器部门）；Human-Computer Interaction group at Hamburg University（汉堡大学人机交互小组）
University of Tsukuba（筑波大学）；Universite de Picardie Jules Verne（皮卡第儒勒·凡尔纳大学）；National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）
In immersive humanoid robot teleoperation, there are three main shortcomings that can alter the transparency of the visual feedback: the lag between the motion of the operator's and robot's head due to network communication delays or slow robot joint motion. This latency could cause a noticeable delay in the visual feedback, which jeopardizes the embodiment quality, can cause dizziness, and affects the interactivity resulting in operator frequent motion pauses for the visual feedback to settle; (ii) the mismatch between the camera's and the headset's field-of-views (FOV), the former having gen…
Dexterous manipulation with anthropomorphic robot hands remains a challenging problem in robotics because of the high-dimensional state and action spaces and complex contacts. Nevertheless, skillful closed-loop manipulation is required to enable humanoid robots to operate in unstructured real-world environments. Reinforcement learning (RL) has traditionally imposed enormous interaction data requirements for optimizing such complex control problems. We introduce a new framework that leverages recent advances in GPU-based simulation along with the strength of imitation learning in guiding policy…
Autonomous Intelligent Systems group of University of Bonn（波恩大学自主智能系统小组）
We present a method for real-time stereo scene capture and remote VR visualization that allows a human operator to freely move their head and thus intuitively control their perspective during teleoperation. The stereo camera is mounted on a 6D robotic arm, which follows the operator's head pose. Existing VR teleoperation systems either induce high latencies on head movements, leading to motion sickness, or use scene reconstruction methods to allow re-rendering of the scene from different perspectives, which cannot handle dynamic scenes effectively. Instead, we present a decoupled approach whic…
The Role of Embodiment in Intuitive Whole-Body Teleoperation for Mobile Manipulation
Computer Science Department, TU Darmstadt（图恩-达姆施塔特大学计算机科学系）；University of São Paulo（圣保罗大学）；Centre for Cognitive Science, TU Darmstadt（图恩-达姆施塔特大学认知科学中心）；Systems AI for Robot Learning, German Research Center for AI (DFKI)（德国人工智能研究中心（DFKI）机器人学习系统AI）
A fundamental challenge in teaching robots is to provide an effective interface for human teachers to demonstrate useful skills to a robot. This challenge is exacerbated in dexterous manipulation, where teaching high-dimensional, contact-rich behaviors often require esoteric teleoperation tools. In this work, we present Holo-Dex, a framework for dexterous manipulation that places a teacher in an immersive mixed reality through commodity VR headsets. The high-fidelity hand pose estimator onboard the headset is used to teleoperate the robot and collect demonstrations for a variety of general-pur…
Robotic avatars promise immersive teleoperation with human-like manipulation and communication capabilities. We present such an avatar system, based on the key components of immersive 3D visualization and transparent force-feedback telemanipulation. Our avatar robot features an anthropomorphic bimanual arm configuration with dexterous hands. The remote human operator drives the arms and fingers through an exoskeleton-based operator station, which provides force feedback both at the wrist and for each finger. The robot torso is mounted on a holonomic base, providing locomotion capability in typ…
Imperial College London（伦敦帝国理工学院）；Indonesian Institute of Sciences - LIPI（印度尼西亚科学院）
In this work, we propose a novel mobile rescue robot equipped with an immersive stereoscopic teleperception and a teleoperation control. This robot is designed with the capability to perform safely a casualty-extraction procedure. We have built a proof-of-concept mobile rescue robot called ResQbot for the experimental platform. An approach called "loco-manipulation" is used to perform the casualty-extraction procedure using the platform. The performance of this robot is evaluated in terms of task accomplishment and safety by conducting a mock rescue experiment. We use a custom-made human-sized…
VR-DAgger: Immersive VR for Dexterous Data Collection and Uncertainty-Guided On-Policy Correction
Robotics Systems Lab（机器人系统实验室）；ETH Zürich（苏黎世联邦理工学院）；ETH AI Center（ETH人工智能中心）；ETH Augmented Reality Research Lab（ETH增强现实研究实验室）；ETH Mobility Initiative（ETH移动性倡议）；ANYbotics AG（ANYbotics公司）；Swiss Federal Railways（瑞士联邦铁路）
