In this paper, we present a multimodal mobile teleoperation system that consists of a novel vision-based hand pose regression network (Transteleop) and an IMU-based arm tracking method. Transteleop observes the human hand through a low-cost depth camera and generates not only joint angles but also depth images of paired robot hand poses through an image-to-image translation process. A keypoint-based reconstruction loss explores the resemblance in appearance and anatomy between human and robotic hands and enriches the local features of reconstructed images. A wearable camera holder enables simultaneous hand-arm control and facilitates the mobility of the whole teleoperation system. Network evaluation results on a test dataset and a variety of complex manipulation tasks that go beyond simple pick-and-place operations show the efficiency and stability of our multimodal teleoperation system.
University of Leeds（利兹大学）；University College London（伦敦大学学院）
Human life is invaluable. When dangerous or life-threatening tasks need to be completed, robotic platforms could be ideal in replacing human operators. Such a task that we focus on in this work is the Explosive Ordnance Disposal. Robot telepresence has the potential to provide safety solutions, given that mobile robots have shown robust capabilities when operating in several environments. However, autonomy may be challenging and risky at this stage, compared to human operation. Teleoperation could be a compromise between full robot autonomy and human presence. In this paper, we present a relat…
A Vision-Based Shared-Control Teleoperation Scheme for Controlling the Robotic Arm of a Four-Legged Robot
Department of Mechanical Engineering, University of São Paulo（机械工程系，圣保罗大学）；Instituto Israelita de Ensino e Pesquisa Albert Einstein, Hospital Israelita Albert Einstein（艾利哈德研究所，以色列医院）
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）
Drone application for aerial manipulation is tested in such areas as industrial maintenance, supporting the rescuers in emergencies, and e-commerce. Most of such applications require teleoperation. The operator receives visual feedback from the camera installed on a robot arm or drone. As aerial manipulation requires delicate and precise motion of robot arm, the camera data delay, narrow field of view, and blurred images caused by drone dynamics can lead the UAV to crash. The paper focuses on the development of a novel teleoperation system for aerial manipulation using Virtual Reality (VR). Th…
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院 (EPFL)）；ETHZ（苏黎世联邦理工学院）；Swiss Federal Institute for Forest Snow, and Landscape Research (WSL)（瑞士联邦森林、雪与景观研究所 (WSL)）
Gesture-based interfaces are often used to achieve a more natural and intuitive teleoperation of robots. Yet, sometimes, gesture control requires postures or movements that cause significant fatigue to the user. In a previous user study, we demonstrated that naïve users can control a fixed-wing drone with torso movements while their arms are spread out. However, this posture induced significant arm fatigue. In this work, we present a passive arm support that compensates the arm weight with a mean torque error smaller than 0.005 N/kg for more than 97% of the range of motion used by subjects to …
One of the goals in telepresence is to be able to perform daily tasks remotely. A key requirement for this is a robust and reliable mobile robotic platform. Ideally, such a platform should support 360-degree stereoscopic vision and semi-autonomous telemanipulation ability. In this technical report, we present our latest work on designing the telepresence mobile robot platform called MAVI. MAVI is a low-cost and robust but extendable platform for research and educational purpose, especially for machine vision and human interaction in telepresence setups. The MAVI platform offers a balance betwe…
Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion
Department of Artificial Intelligence, Korea University（人工智能系，韩国大学）；Department of Computer Science, RPTU Kaiserslautern-Landau（计算机科学系，RPTU凯撒斯劳滕-兰道）；Embedded Intelligence, German Research Center for Artificial Intelligence (DFKI)（嵌入式智能，德国人工智能研究中心（DFKI））
Teleoperation provides a way for human operators to guide robots in situations where full autonomy is challenging or where direct human intervention is required. It can also be an important tool to teach robots in order to achieve autonomous behaviour later on. The increased availability of collaborative robot arms and Virtual Reality (VR) devices provides ample opportunity for development of novel teleoperation methods. Since robot arms are often kinematically different from human arms, mapping human motions to a robot in real-time is not trivial. Additionally, a human operator might steer th…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Teleoperation (i.e., controlling a robot with human motion) proves promising in enabling a humanoid robot to move as dynamically as a human. But how to map human motion to a humanoid robot matters because a human and a humanoid robot rarely have identical topologies and dimensions. This work presents an experimental study that utilizes reaction tests to compare the proposed joint space mapping and the proposed task space mapping for dynamic teleoperation of an anthropomorphic robotic arm that possesses human-level dynamic motion capabilities. The experimental results suggest that the robot ach…
HATS: A Human-Agent Teleoperation System for Multi-Arm Data Collection
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Nanyang Technological University（南洋理工大学）；Imperial College London（帝国理工学院）
Teleoperation offers the possibility of imparting robotic systems with sophisticated reasoning skills, intuition, and creativity to perform tasks. However, current teleoperation solutions for high degree-of-actuation (DoA), multi-fingered robots are generally cost-prohibitive, while low-cost offerings usually provide reduced degrees of control. Herein, a low-cost, vision based teleoperation system, DexPilot, was developed that allows for complete control over the full 23 DoA robotic system by merely observing the bare human hand. DexPilot enables operators to carry out a variety of complex man…
Dept. of Mechanical Engineering（机械工程系）；Dept. of Computer Science, Johns Hopkins University（约翰斯·霍普金斯大学计算机科学系）
Teleoperation platforms often require the user to be situated at a fixed location to both visualize and control the movement of the robot and thus do not provide the operator with much mobility. One example is in existing robotic surgery solutions that require the surgeons to be away from the patient, attached to consoles where their heads must be fixed and their arms can only move in a limited space. This creates a barrier between physicians and patients that does not exist in normal surgery. To address this issue, we propose a mobile telesurgery solution where the surgeons are no longer mech…
IMMERTWIN: A Mixed Reality Framework for Enhanced Robotic Arm Teleoperation
School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）；ICPS Research Center, National Institute of Advanced Industrial Science and Technology (AIST)（日本国立先进工业科学技术研究院ICPS研究中心）
DexTele: A Dual-Arm Dexterous Teleoperation System Based on Motion Retargeting and Adaptive Force Control
School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen（人工智能学院，香港中文大学（深圳））；School of Computing, University of Portsmouth（计算学院，朴茨茅斯大学）
TeleOpBench: A Simulator-Centric Benchmark for Dual-Arm Dexterous Teleoperation
Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The University of Hong Kong（香港大学）；Feeling AI（FeelAI）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）
KUKAloha: A General, Low-Cost, and Shared-Control based Teleoperation Framework for Construction Robot Arm
Department of Civil and Environmental Engineering, University of Michigan, United States of America（密歇根大学土木与环境工程系）；Department of Robotics, University of Michigan, United States of America（密歇根大学机器人系）
Affordance-Guided Dual-Armed Disassembly Teleoperation for Mating Parts
Department of Systems Innovation, Graduate School of Engineering Science, Osaka University（大阪大学系统创新系）；Industrial Cyber-physical Systems Research Center, The National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）；Manufacturing Innovation Division, Panasonic Holdings Corporation（松下电器控股公司）
This paper discusses the feasibility of using configuration space (C-space) as a means of visualization and control in operator-guided real-time motion of a robot arm manipulator. The motivation is to improve performance of the human operator in tasks involving the manipulator motion in an environment with obstacles. Unlike some other motion planning tasks, operators are known to make expensive mistakes in such tasks, even in a simpler two-dimensional case. They have difficulty learning better procedures and their performance improves very little with practice. Using an example of a two-dimens…
Imitation Learning (IL) is a powerful paradigm to teach robots to perform manipulation tasks by allowing them to learn from human demonstrations collected via teleoperation, but has mostly been limited to single-arm manipulation. However, many real-world tasks require multiple arms, such as lifting a heavy object or assembling a desk. Unfortunately, applying IL to multi-arm manipulation tasks has been challenging -- asking a human to control more than one robotic arm can impose significant cognitive burden and is often only possible for a maximum of two robot arms. To address these challenges,…
German Aerospace Center（德国航空航天中心）；Hamburg University（汉堡大学）；Technical University of Munich（慕尼黑工业大学）
Telerobotic systems must adapt to new environmental conditions and deal with high uncertainty caused by long-time delays. As one of the best alternatives to human-level intelligence, Reinforcement Learning (RL) may offer a solution to cope with these issues. This paper proposes to integrate RL with the Model Mediated Teleoperation (MMT) concept. The teleoperator interacts with a simulated virtual environment, which provides instant feedback. Whereas feedback from the real environment is delayed, feedback from the model is instantaneous, leading to high transparency. The MMT is realized in comb…
Istituto Italiano di Tecnologia（意大利理工学院）；Universita di Pisa（比萨大学）
Teleoperation is a widely adopted strategy to control robotic manipulators executing complex tasks that require highly dexterous movements and critical high-level intelligence. Classical teleoperation schemes are based on either joystick control, or on more intuitive interfaces which map directly the user arm motions into one robot arm's motions. These approaches have limits when the execution of a given task requires reconfigurable multiple robotic arm systems. Indeed, the simultaneous teleoperation of two or more robot arms could extend the workspace of the manipulation cell, or increase its…
This paper explores the possibility of improving bilateral robot manipulation task performance through optimizing the robot morphology and configuration of the system through motion. To optimize the design for different scenarios, we select a set of tasks that represent the variability in small scale manipulation (e.g. pick and place, tasks involving positioning and orientation) and track the motion to obtain a reproducible trajectory. Kinematic data is captured through an electromagnetic (EM) tracker system while a human subject performs the tasks. Then, the data is pre-processed and used to …
Northwestern University（西北大学）；Shirley Ryan AbilityLab（雪莉·瑞安能力实验室）
Assistive robotic devices can increase the independence of individuals with motor impairments. However, each person is unique in their level of injury, preferences, and skills, which moreover can change over time. Further, the amount of assistance required can vary throughout the day due to pain or fatigue, or over longer periods due to rehabilitation, debilitating conditions, or aging. Therefore, in order to become an effective team member, the assistive machine should be able to learn from and adapt to the human user. To do so, we need to be able to characterize the user's control commands t…
Imperial College of Science Technology and Medicine（帝国理工学院）；Nanyang Technological University（南洋理工大学）
Many teleoperation tasks require three or more tools working together, which need the cooperation of multiple operators. The effectiveness of such schemes may be limited by communication. Trimanipulation by a single operator using an artificial third arm controlled together with their natural arms is a promising solution to this issue. Foot-controlled interfaces have previously shown the capability to be used for the continuous control of robot arms. However, the use of such interfaces for controlling a supernumerary robotic limb (SRLs) in coordination with the natural limbs, is not well under…
