Teleoperation (i.e., controlling a robot with human motion) proves promising in enabling a humanoid robot to move as dynamically as a human. But how to map human motion to a humanoid robot matters because a human and a humanoid robot rarely have identical topologies and dimensions. This work presents an experimental study that utilizes reaction tests to compare the proposed joint space mapping and the proposed task space mapping for dynamic teleoperation of an anthropomorphic robotic arm that possesses human-level dynamic motion capabilities. The experimental results suggest that the robot achieved similar and, in some cases, human-level dynamic performances with both mappings for the six participating human subjects. All subjects became proficient at teleoperating the robot with both mappings after practice, despite that the subjects and the robot differed in size and link length ratio and that the teleoperation required the subjects to move unintuitively. Yet, most subjects developed their teleoperation proficiencies more quickly with the task space mapping than with the joint space mapping after similar amounts of practice. This study also indicates the potential values of a three-dimensional task space mapping, a teleoperation training simulator, and force feedback to the human pilot for intuitive and dynamic teleoperation of a humanoid robot's arms.
We present a method for EMG-driven teleoperation of non-anthropomorphic robot hands. EMG sensors are appealing as a wearable, inexpensive, and unobtrusive way to gather information about the teleoperator's hand pose. However, mapping from EMG signals to the pose space of a non-anthropomorphic hand presents multiple challenges. We present a method that first projects from forearm EMG into a subspace relevant to teleoperation. To increase robustness, we use a model which combines continuous and discrete predictors along different dimensions of this subspace. We then project from the teleoperatio…
Teleoperation is a valuable tool for robotic manipulators in highly unstructured environments. However, finding an intuitive mapping between a human hand and a non-anthropomorphic robot hand can be difficult, due to the hands' dissimilar kinematics. In this paper, we seek to create a mapping between the human hand and a fully actuated, non-anthropomorphic robot hand that is intuitive enough to enable effective real-time teleoperation, even for novice users. To accomplish this, we propose a low-dimensional teleoperation subspace which can be used as an intermediary for mapping between hand pose…
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness and thus create full immersion, as well as offering the operator suitable control interfaces to achieve efficient and robust task fulfillment. We present a bimanual telemanipulation system consisting…
Human-in-the-loop manipulation is useful in when autonomous grasping is not able to deal sufficiently well with corner cases or cannot operate fast enough. Using the teleoperator's hand as an input device can provide an intuitive control method but requires mapping between pose spaces which may not be similar. We propose a low-dimensional and continuous teleoperation subspace which can be used as an intermediary for mapping between different hand pose spaces. We present an algorithm to project between pose space and teleoperation subspace. We use a non-anthropomorphic robot to experimentally p…
ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling
School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）人工智能学院）；School of Computing, University of Portsmouth（朴茨茅斯大学计算学院）
Teleoperation provides a way for human operators to guide robots in situations where full autonomy is challenging or where direct human intervention is required. It can also be an important tool to teach robots in order to achieve autonomous behaviour later on. The increased availability of collaborative robot arms and Virtual Reality (VR) devices provides ample opportunity for development of novel teleoperation methods. Since robot arms are often kinematically different from human arms, mapping human motions to a robot in real-time is not trivial. Additionally, a human operator might steer th…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
The operation of telerobotic systems can be a challenging task, requiring intuitive and efficient interfaces to enable inexperienced users to attain a high level of proficiency. Body-Machine Interfaces (BoMI) represent a promising alternative to standard control devices, such as joysticks, because they leverage intuitive body motion and gestures. It has been shown that the use of Virtual Reality (VR) and first-person view perspectives can increase the user's sense of presence in avatars. However, it is unclear if these beneficial effects occur also in the teleoperation of non-anthropomorphic r…
HATS: A Human-Agent Teleoperation System for Multi-Arm Data Collection
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Nanyang Technological University（南洋理工大学）；Imperial College London（帝国理工学院）
DexTele: A Dual-Arm Dexterous Teleoperation System Based on Motion Retargeting and Adaptive Force Control
School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen（人工智能学院，香港中文大学（深圳））；School of Computing, University of Portsmouth（计算学院，朴茨茅斯大学）
KUKAloha: A General, Low-Cost, and Shared-Control based Teleoperation Framework for Construction Robot Arm
Department of Civil and Environmental Engineering, University of Michigan, United States of America（密歇根大学土木与环境工程系）；Department of Robotics, University of Michigan, United States of America（密歇根大学机器人系）
Affordance-Guided Dual-Armed Disassembly Teleoperation for Mating Parts
Department of Systems Innovation, Graduate School of Engineering Science, Osaka University（大阪大学系统创新系）；Industrial Cyber-physical Systems Research Center, The National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）；Manufacturing Innovation Division, Panasonic Holdings Corporation（松下电器控股公司）
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）
Drone application for aerial manipulation is tested in such areas as industrial maintenance, supporting the rescuers in emergencies, and e-commerce. Most of such applications require teleoperation. The operator receives visual feedback from the camera installed on a robot arm or drone. As aerial manipulation requires delicate and precise motion of robot arm, the camera data delay, narrow field of view, and blurred images caused by drone dynamics can lead the UAV to crash. The paper focuses on the development of a novel teleoperation system for aerial manipulation using Virtual Reality (VR). Th…
Advantages of Multimodal versus Verbal-Only Robot-to-Human Communication with an Anthropomorphic Robotic Mock Driver
Centre for Applied Autonomous Sensor Systems (AASS)（应用自主传感器系统中心（AASS））；Örebro University（厄勒布鲁大学）；Robert Bosch GmbH（罗伯特·博世有限公司）；University of Skövde（舍夫德大学）；TU Munich（慕尼黑工业大学）
Suction Leap-Hand: Suction Cups on a Multi-fingered Hand Enable Embodied Dexterity and In-Hand Teleoperation
School of Informatics, University of Edinburgh（信息学院，爱丁堡大学）；School of Physics, Engineering and Technology, University of York（物理、工程与技术学院，约克大学）；Tsinghua University（清华大学）
Ben-Gurion University of the Negev（内盖夫本-古里安大学）；Technical University Darmstadt（达姆施塔特工业大学）
The development of autonomous robotic systems that can learn from human demonstrations to imitate a desired behavior - rather than being manually programmed - has huge technological potential. One major challenge in imitation learning is the correspondence problem: how to establish corresponding states and actions between expert and learner, when the embodiments of the agents are different (morphology, dynamics, degrees of freedom, etc.). Many existing approaches in imitation learning circumvent the correspondence problem, for example, kinesthetic teaching or teleoperation, which are performed…
Universität Hamburg（汉堡大学）；RWTH Aachen University（亚琛工业大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Tsinghua University（清华大学）
In this paper, we present a multimodal mobile teleoperation system that consists of a novel vision-based hand pose regression network (Transteleop) and an IMU-based arm tracking method. Transteleop observes the human hand through a low-cost depth camera and generates not only joint angles but also depth images of paired robot hand poses through an image-to-image translation process. A keypoint-based reconstruction loss explores the resemblance in appearance and anatomy between human and robotic hands and enriches the local features of reconstructed images. A wearable camera holder enables simu…
Istituto Italiano di Tecnologia（意大利理工学院）；Universita di Pisa（比萨大学）
Teleoperation is a widely adopted strategy to control robotic manipulators executing complex tasks that require highly dexterous movements and critical high-level intelligence. Classical teleoperation schemes are based on either joystick control, or on more intuitive interfaces which map directly the user arm motions into one robot arm's motions. These approaches have limits when the execution of a given task requires reconfigurable multiple robotic arm systems. Indeed, the simultaneous teleoperation of two or more robot arms could extend the workspace of the manipulation cell, or increase its…
This paper discusses the feasibility of using configuration space (C-space) as a means of visualization and control in operator-guided real-time motion of a robot arm manipulator. The motivation is to improve performance of the human operator in tasks involving the manipulator motion in an environment with obstacles. Unlike some other motion planning tasks, operators are known to make expensive mistakes in such tasks, even in a simpler two-dimensional case. They have difficulty learning better procedures and their performance improves very little with practice. Using an example of a two-dimens…
This paper explores the possibility of improving bilateral robot manipulation task performance through optimizing the robot morphology and configuration of the system through motion. To optimize the design for different scenarios, we select a set of tasks that represent the variability in small scale manipulation (e.g. pick and place, tasks involving positioning and orientation) and track the motion to obtain a reproducible trajectory. Kinematic data is captured through an electromagnetic (EM) tracker system while a human subject performs the tasks. Then, the data is pre-processed and used to …
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院 (EPFL)）；ETHZ（苏黎世联邦理工学院）；Swiss Federal Institute for Forest Snow, and Landscape Research (WSL)（瑞士联邦森林、雪与景观研究所 (WSL)）
Gesture-based interfaces are often used to achieve a more natural and intuitive teleoperation of robots. Yet, sometimes, gesture control requires postures or movements that cause significant fatigue to the user. In a previous user study, we demonstrated that naïve users can control a fixed-wing drone with torso movements while their arms are spread out. However, this posture induced significant arm fatigue. In this work, we present a passive arm support that compensates the arm weight with a mean torque error smaller than 0.005 N/kg for more than 97% of the range of motion used by subjects to …
Northwestern University（西北大学）；Shirley Ryan AbilityLab（雪莉·瑞安能力实验室）
Assistive robotic devices can increase the independence of individuals with motor impairments. However, each person is unique in their level of injury, preferences, and skills, which moreover can change over time. Further, the amount of assistance required can vary throughout the day due to pain or fatigue, or over longer periods due to rehabilitation, debilitating conditions, or aging. Therefore, in order to become an effective team member, the assistive machine should be able to learn from and adapt to the human user. To do so, we need to be able to characterize the user's control commands t…
Dept. of Mechanical Engineering（机械工程系）；Dept. of Computer Science, Johns Hopkins University（约翰斯·霍普金斯大学计算机科学系）
Teleoperation platforms often require the user to be situated at a fixed location to both visualize and control the movement of the robot and thus do not provide the operator with much mobility. One example is in existing robotic surgery solutions that require the surgeons to be away from the patient, attached to consoles where their heads must be fixed and their arms can only move in a limited space. This creates a barrier between physicians and patients that does not exist in normal surgery. To address this issue, we propose a mobile telesurgery solution where the surgeons are no longer mech…
Teleoperation offers the possibility of imparting robotic systems with sophisticated reasoning skills, intuition, and creativity to perform tasks. However, current teleoperation solutions for high degree-of-actuation (DoA), multi-fingered robots are generally cost-prohibitive, while low-cost offerings usually provide reduced degrees of control. Herein, a low-cost, vision based teleoperation system, DexPilot, was developed that allows for complete control over the full 23 DoA robotic system by merely observing the bare human hand. DexPilot enables operators to carry out a variety of complex man…
IMMERTWIN: A Mixed Reality Framework for Enhanced Robotic Arm Teleoperation
School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）；ICPS Research Center, National Institute of Advanced Industrial Science and Technology (AIST)（日本国立先进工业科学技术研究院ICPS研究中心）
Robotic assistance in robot arm teleoperation tasks has recently gained a lot of traction in industrial and domestic environment. A wide variety of input devices is used in such setups. Due to the noise in the input signals (e.g., Brain Computer Interface (BCI)) or delays due to environmental conditions (e.g., space robot teleoperation), users need assistive autonomy that keeps them in control while following predefined trajectories and avoids obstacles. This assistance calls for activity representations that are easy to define by the operator and able to take the dynamic world state into cons…
Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks by allowing them to learn from human demonstrations collected via teleoperation, but has mostly been limited to single-arm manipulation. However, many real-world tasks require multiple arms, such as lifting a heavy object or assembling a desk. Unfortunately, applying IL to multi-arm manipulation tasks has been challenging -- asking a human to control more than one robotic arm can impose significant cognitive burden and is often only possible for a maximum of two robot arms. To address these challenges,…
A Vision-Based Shared-Control Teleoperation Scheme for Controlling the Robotic Arm of a Four-Legged Robot
Department of Mechanical Engineering, University of São Paulo（机械工程系，圣保罗大学）；Instituto Israelita de Ensino e Pesquisa Albert Einstein, Hospital Israelita Albert Einstein（艾利哈德研究所，以色列医院）
TeleOpBench: A Simulator-Centric Benchmark for Dual-Arm Dexterous Teleoperation
Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The University of Hong Kong（香港大学）；Feeling AI（FeelAI）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）
