Mapping operator motions to a robot is a key problem in teleoperation. Due to differences between workspaces, such as object locations, it is particularly challenging to derive smooth motion mappings that fulfill different goals (e.g. picking objects with different poses on the two sides or passing through key points). Indeed, most state-of-the-art methods rely on mode switches, leading to a discontinuous, low-transparency experience. In this paper, we propose a unified formulation for position, orientation and velocity mappings based on the poses of objects of interest in the operator and robot workspaces. We apply it in the context of bilateral teleoperation. Two possible implementations to achieve the proposed mappings are studied: an iterative approach based on locally-weighted translations and rotations, and a neural network approach. Evaluations are conducted both in simulation and using two torque-controlled Franka Emika Panda robots. Our results show that, despite longer training times, the neural network approach provides faster mapping evaluations and lower interaction forces for the operator, which are crucial for continuous, real-time teleoperation.
Attaching a robotic manipulator to a flying base allows for significant improvements in the reachability and versatility of manipulation tasks. In order to explore such systems while taking advantage of human capabilities in terms of perception and cognition, bilateral teleoperation arises as a reasonable solution. However, since most telemanipulation tasks require visual feedback in addition to the haptic one, real-time (task-dependent) positioning of a video camera, which is usually attached to the flying base, becomes an additional objective to be fulfilled. Since the flying base is part of…
Composite Adaptive Control for Bilateral Teleoperation Systems without Persistency of Excitation
School of Automation and Electrical Engineering, University of Science and Technology Beijing（自动化与电气工程学院，北京科技大学）；Department of Automatic Control, Lund University, P.O. Box 118, 22100 Lund, Sweden（自动控制系，卢德大学）
Design and Experimental Validation of Sensorless 4-Channel Bilateral Teleoperation for Low-Cost Manipulators
Degree Programs in Intelligent and Mechanical Interaction Systems, University of Tsukuba（智能与机械交互系统专业，东京大学）；Faculty of Engineering, Information and Systems, University of Tsukuba（工程、信息与系统学部，东京大学）；Department of Electrical Engineering, Electronics, and Applied Physics, Saitama University（电子工程、电子学与应用物理系，埼玉大学）
Bilateral teleoperation offers an intriguing solution towards shared autonomy with aerial vehicles in contact-based inspection and manipulation tasks. Omnidirectional aerial robots allow for full pose operations, making them particularly attractive in such tasks. Naturally, the question arises whether standard bilateral teleoperation methodologies are suitable for use with these vehicles. In this work, a fully decoupled 6DoF bilateral teleoperation framework for aerial physical interaction is designed and tested for the first time. The method is based on the well established rate control, rece…
Graduate School of Engineering Science, Osaka University（大阪大学工学研究科）；City University of Hongkong（香港城市大学）
We present bilateral teleoperation system for task learning and robot motion generation. Our system includes a bilateral teleoperation platform and a deep learning software. The deep learning software refers to human demonstration using the bilateral teleoperation platform to collect visual images and robotic encoder values. It leverages the datasets of images and robotic encoder information to learn about the inter-modal correspondence between visual images and robot motion. In detail, the deep learning software uses a combination of Deep Convolutional Auto-Encoders (DCAE) over image regions,…
Input-gated Bilateral Teleoperation: An Easy-to-implement Force Feedback Teleoperation Method for Low-cost Hardware
Research & Development Group, Hitachi, Ltd.（日立株式会社研发部）；Department of Intermedia Art and Science School of Fundamental Science and Engineering, Waseda University（早稻田大学基础科学与工程学部多媒体艺术与科学系）
This paper develops a novel force observer for bilateral teleoperation systems. Type-2 fuzzy logic is used to describe the overall dynamic system, and Moving Horizon Estimation (MHE) is employed to assess clean states as well as the values of dynamic uncertainties, and simultaneously filter out the measurement noises, which guarantee the high degree of accuracy for the observed forces. Compared with the existing methods, the proposed force observer can run without knowing exact mathematical dynamic functions and is robust to different kinds of noises. A force-reflection four-channel teleoperat…
Tampere University（坦佩雷大学）；Canadian Space Agency（加拿大空间局）
This study designs a high-precision bilateral teleoperation control for a dissimilar master-slave system. The proposed nonlinear control design takes advantage of a novel subsystem-dynamics-based control method that allows designing of individual (decentralized) model-based controllers for the manipulators locally at the subsystem level. Very importantly, a dynamic model of the human operator is incorporated into the control of the master manipulator. The individual controllers for the dissimilar master and slave manipulators are connected in a specific communication channel for the bilateral …
This paper explores the possibility of improving bilateral robot manipulation task performance through optimizing the robot morphology and configuration of the system through motion. To optimize the design for different scenarios, we select a set of tasks that represent the variability in small scale manipulation (e.g. pick and place, tasks involving positioning and orientation) and track the motion to obtain a reproducible trajectory. Kinematic data is captured through an electromagnetic (EM) tracker system while a human subject performs the tasks. Then, the data is pre-processed and used to …
LETS: Low-Overhead External-Torque Sensing for Force-Feedback Bilateral Teleoperation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Beijing Amigos Robot Technology Co., Ltd.（北京阿米格斯机器人科技有限公司）
Haptic bilateral teleoperation system for free-hand dental procedures
Department of Excellence 2023/2027 Project（卓越部门2023/2027项目）；Department of Information Engineering, Electrical Engineering and Applied Mathematics (DIEM), University of Salerno（信息工程、电气工程和应用数学系（DIEM），萨勒诺大学）；Specialization School in Oral Surgery, University of Naples Federico II（口腔外科专业学校，那不勒斯费德里科二世大学）
Safety-Critical Bilateral Teleoperation for Omnidirectional Aerial Manipulation Using Force-Sensorless Haptic Feedback
Seoul National University（首尔大学）；Automation and System Research Institute (ASRI)（自动化与系统研究所）；Institute of Advanced Aerospace Technology (IAAT)（先进航空航天技术研究所）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carnegie Mellon University（卡内基梅隆大学）
Bilateral teleoperation provides humanoid robots with human planning intelligence while enabling the human to feel what the robot feels. It has the potential to transform physically capable humanoid robots into dynamically intelligent ones. However, dynamic bilateral locomotion teleoperation remains as a challenge due to the complex dynamics it involves. This work presents our initial step to tackle this challenge via the concept of wheeled humanoid robot locomotion teleoperation by body tilt. Specifically, we developed a force-feedback-capable whole-body human-machine interface (HMI), and des…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Teleoperation (i.e., controlling a robot with human motion) proves promising in enabling a humanoid robot to move as dynamically as a human. But how to map human motion to a humanoid robot matters because a human and a humanoid robot rarely have identical topologies and dimensions. This work presents an experimental study that utilizes reaction tests to compare the proposed joint space mapping and the proposed task space mapping for dynamic teleoperation of an anthropomorphic robotic arm that possesses human-level dynamic motion capabilities. The experimental results suggest that the robot ach…
Teleoperation is a valuable tool for robotic manipulators in highly unstructured environments. However, finding an intuitive mapping between a human hand and a non-anthropomorphic robot hand can be difficult, due to the hands' dissimilar kinematics. In this paper, we seek to create a mapping between the human hand and a fully actuated, non-anthropomorphic robot hand that is intuitive enough to enable effective real-time teleoperation, even for novice users. To accomplish this, we propose a low-dimensional teleoperation subspace which can be used as an intermediary for mapping between hand pose…
Resource-constrained mobile robots that lack the capability to be completely autonomous can rely on a human or AI supervisor acting at a remote site (e.g., control station or cloud) for their control. Such a supervised autonomy or cloud-based control of a robot poses high networking and computing capabilities requirements at both sites, which are not easy to achieve. This paper introduces and analyzes a new analog twin framework by synchronizing mobility between two mobile robots, where one robot acts as an analog twin to the other robot. We devise a novel priority-based supervised bilateral t…
Human-in-the-loop manipulation is useful in when autonomous grasping is not able to deal sufficiently well with corner cases or cannot operate fast enough. Using the teleoperator's hand as an input device can provide an intuitive control method but requires mapping between pose spaces which may not be similar. We propose a low-dimensional and continuous teleoperation subspace which can be used as an intermediary for mapping between different hand pose spaces. We present an algorithm to project between pose space and teleoperation subspace. We use a non-anthropomorphic robot to experimentally p…
Purpose: Recent developments in robotics and artificial intelligence (AI) have led to significant advances in healthcare technologies enhancing robot-assisted minimally invasive surgery (RAMIS) in some surgical specialties. However, current human-robot interfaces lack intuitive teleoperation and cannot mimic surgeon's hand/finger sensing and fine motion. These limitations make tele-operated robotic surgery not suitable for micro-surgery and difficult to learn for established surgeons. We report a pilot study showing an intuitive way of recording and mapping surgeon's gross hand motion and the …
Although data-driven motion mapping methods are promising to allow intuitive robot control and teleoperation that generate human-like robot movement, they normally require tedious pair-wise training for each specific human and robot pair. This paper proposes a transferability-based mapping scheme to allow new robot and human input systems to leverage the mapping of existing trained pairs to form a mapping transfer chain, which will reduce the number of new pair-specific mappings that need to be generated. The first part of the mapping schematic is the development of a Synergy Mapping via Dual-…
Failure-Aware Bimanual Teleoperation via Conservative Value Guided Assistance
School of Advanced Engineering, Great Bay University（先进工程学院，大湾大学）；School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；Department of Mechanical Engineering, The Hong Kong Polytechnic University（机械工程系，香港理工大学）；College of Mechanical and Electrical Engineering, Northeast Forestry University（机械电子工程学院，东北林业大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电气与电子工程学院，南洋理工大学）
We present a method for EMG-driven teleoperation of non-anthropomorphic robot hands. EMG sensors are appealing as a wearable, inexpensive, and unobtrusive way to gather information about the teleoperator's hand pose. However, mapping from EMG signals to the pose space of a non-anthropomorphic hand presents multiple challenges. We present a method that first projects from forearm EMG into a subspace relevant to teleoperation. To increase robustness, we use a model which combines continuous and discrete predictors along different dimensions of this subspace. We then project from the teleoperatio…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Robotic systems that can dynamically combine manipulation and locomotion could facilitate dangerous or physically demanding labor. For instance, firefighter humanoid robots could leverage their body by leaning against collapsed building rubble to push it aside. Here we introduce a teleoperation system that targets the realization of these tasks using human whole-body motor skills. We describe a new wheeled humanoid platform, SATYRR, and a novel hands-free teleoperation architecture using a whole-body Human Machine Interface (HMI). This system enables telelocomotion of the humanoid robot using …
Investigating the Benefits of Nonlinear Action Maps in Data-Driven Teleoperation
University of Alberta（阿尔伯塔大学）；Universite de Montreal（蒙特利尔大学）；University of Innsbruck（因斯布鲁克大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））
Institute of Perception, Action and Behaviour, School of Informatics, The University of Edinburgh（爱丁堡大学信息学院感知、行动与行为研究所）；Oxford Robotics Institute at the University of Oxford（牛津大学牛津机器人研究所）；School of Computer Science/Department of Electrical and Computer Engineering at McGill University（麦吉尔大学计算机科学学院/电气与计算机工程系）
This paper presents a control framework to teleoperate a quadruped robot's foot for operator-guided haptic exploration of the environment. Since one leg of a quadruped robot typically only has 3 actuated degrees of freedom (DoFs), the torso is employed to assist foot posture control via a hierarchical whole-body controller. The foot and torso postures are controlled by two analytical Cartesian impedance controllers cascaded by a null space projector. The contact forces acting on supporting feet are optimized by quadratic programming (QP). The foot's Cartesian impedance controller may also esti…
Teleoperation provides a way for human operators to guide robots in situations where full autonomy is challenging or where direct human intervention is required. It can also be an important tool to teach robots in order to achieve autonomous behaviour later on. The increased availability of collaborative robot arms and Virtual Reality (VR) devices provides ample opportunity for development of novel teleoperation methods. Since robot arms are often kinematically different from human arms, mapping human motions to a robot in real-time is not trivial. Additionally, a human operator might steer th…
University of California, Berkeley（加州大学伯克利分校）；Embodied Intelligence；OpenAI；International Computer Science Institute (ICSI)（国际计算机科学研究所（ICSI））
Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor ski…
