Drone teleoperation is usually accomplished using remote radio controllers, devices that can be hard to master for inexperienced users. Moreover, the limited amount of information fed back to the user about the robot's state, often limited to vision, can represent a bottleneck for operation in several conditions. In this work, we present a wearable interface for drone teleoperation and its evaluation through a user study. The two main features of the proposed system are a data glove to allow the user to control the drone trajectory by hand motion and a haptic system used to augment their awareness of the environment surrounding the robot. This interface can be employed for the operation of robotic systems in line of sight (LoS) by inexperienced operators and allows them to safely perform tasks common in inspection and search-and-rescue missions such as approaching walls and crossing narrow passages with limited visibility conditions. In addition to the design and implementation of the wearable interface, we performed a systematic study to assess the effectiveness of the system through three user studies (n = 36) to evaluate the users' learning path and their ability to perform tasks with limited visibility. We validated our ideas in both a simulated and a real-world environment. Our results demonstrate that the proposed system can improve teleoperation performance in different cases compared to standard remote controllers, making it a viable alternative to standard Human-Robot Interfaces.
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）
Drone application for aerial manipulation is tested in such areas as industrial maintenance, supporting the rescuers in emergencies, and e-commerce. Most of such applications require teleoperation. The operator receives visual feedback from the camera installed on a robot arm or drone. As aerial manipulation requires delicate and precise motion of robot arm, the camera data delay, narrow field of view, and blurred images caused by drone dynamics can lead the UAV to crash. The paper focuses on the development of a novel teleoperation system for aerial manipulation using Virtual Reality (VR). Th…
People learn motor activities best when they are conscious of their errors and make a concerted effort to correct them. While haptic interfaces can facilitate motor training, existing interfaces are often bulky and do not always ensure post-training skill retention. Here, we describe a programmable haptic sleeve composed of textile-based electroadhesive clutches for skill acquisition and retention. We show its functionality in a motor learning study where users control a drone's movement using elbow joint rotation. Haptic feedback is used to restrain elbow motion and make users aware of their …
This paper presents a novel approach to haptic teleoperation. Specifically, we use control barrier functions (CBFs) to generate force feedback to help human operators safely fly quadrotor UAVs. CBFs take a control signal as input and output a control signal that is as close as possible to the initial control signal, while also meeting specified safety constraints. In the proposed method, we generate haptic force feedback based on the difference between a command issued by the human operator and the safe command returned by a CBF. In this way, if the user issues an unsafe control command, the h…
We present a novel haptic teleoperation approach that considers not only the safety but also the stability of a teleoperation system. Specifically, we build upon previous work on haptic shared control, which uses control barrier functions (CBFs) to generate a reference haptic feedback that informs the human operator on the internal state of the system, helping them to safely navigate the robot without taking away their control authority. Crucially, in this approach the force rendered to the user is not directly reflected in the motion of the robot (which is still directly controlled by the use…
University of Sussex（萨塞克斯大学）；University of Edinburgh（爱丁堡大学）
The teleoperation of robots enables remote intervention in distant and dangerous tasks without putting the operator in harm's way. However, remote operation faces fundamental challenges due to limits in communication delays. The proposed work improves the performances of teleoperation architecture based on Fractal Impedance Controller (FIC) by integrating into the haptic teleoperation pipeline a postural optimisation that also accounts for the replica robots' physical limitations. This update improves dynamic interactions by trading off tracking accuracy to maintain the system within its power…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；University of Southern Denmark（南丹麦大学）；University of Genoa（热那亚大学）
Complex tasks require human collaboration since robots do not have enough dexterity. However, robots are still used as instruments and not as collaborative systems. We are introducing a framework to ensure safety in a human-robot collaborative environment. The system is composed of a haptic feedback display, low-cost wearable mocap, and a new collision avoidance algorithm based on the Artificial Potential Fields (APF). Wearable optical motion capturing system enables tracking the human hand position with high accuracy and low latency on large working areas. This study evaluates whether haptic …
Idiap Research Institute（伊迪亚普研究所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Haptic guidance is a powerful technique to combine the strengths of humans and autonomous systems for teleoperation. The autonomous system can provide haptic cues to enable the operator to perform precise movements; the operator can interfere with the plan of the autonomous system leveraging his/her superior cognitive capabilities. However, providing haptic cues such that the individual strengths are not impaired is challenging because low forces provide little guidance, whereas strong forces can hinder the operator in realizing his/her plan. Based on variational inference, we learn a Gaussian…
University of the West of England（西英格兰大学）；Toshiba Europe Ltd.（东芝欧洲有限公司）
Haptic teleoperation is typically realized through wired networking technologies (e.g., Ethernet) which guarantee performance of control loops closed over the communication medium, particularly in terms of latency, jitter, and reliability. This demonstration shows the capability of conducting haptic teleoperation over a novel low-power wireless control technology, called GALLOP, in a nuclear decommissioning use-case. It shows the viability of GALLOP for meeting latency, timeliness, and safety requirements of haptic teleoperation. Evaluation conducted as part of the demonstration reveals that G…
Shared autonomy teleoperation can guarantee safety, but does so by reducing the human operator's control authority, which can lead to reduced levels of human-robot agreement and user satisfaction. This paper presents a novel haptic shared autonomy teleoperation paradigm that uses haptic feedback to inform the user about the inner state of a shared autonomy paradigm, while still guaranteeing safety. This differs from haptic shared control, which uses haptic feedback to inform the user's actions, but gives the human operator full control over the robot's actions. We conducted a user study in whi…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
We report on the teleoperation system DronePick which provides remote object picking and delivery by a human-controlled quadcopter. The main novelty of the proposed system is that the human user continuously gets the visual and haptic feedback for accurate teleoperation. DronePick consists of a quadcopter equipped with a magnetic grabber, a tactile glove with finger motion tracking sensor, hand tracking system, and the Virtual Reality (VR) application. The human operator teleoperates the quadcopter by changing the position of the hand. The proposed vibrotactile patterns representing the locati…
Safety-Critical Bilateral Teleoperation for Omnidirectional Aerial Manipulation Using Force-Sensorless Haptic Feedback
Seoul National University（首尔大学）；Automation and System Research Institute (ASRI)（自动化与系统研究所）；Institute of Advanced Aerospace Technology (IAAT)（先进航空航天技术研究所）
Force estimation using neural networks is a promising approach to enable haptic feedback in minimally invasive surgical robots without end-effector force sensors. Various network architectures have been proposed, but none have been tested in real time with surgical-like manipulations. Thus, questions remain about the real-time transparency and stability of force feedback from neural network-based force estimates. We characterize the real-time impedance transparency and stability of force feedback rendered on a da Vinci Research Kit teleoperated surgical robot using neural networks with vision-…
Mobile manipulators in the home can provide increased autonomy to individuals with severe motor impairments, who often cannot complete activities of daily living (ADLs) without the help of a caregiver. Teleoperation of an assistive mobile manipulator could enable an individual with motor impairments to independently perform self-care and household tasks, yet limited motor function can impede one's ability to interface with a robot. In this work, we present a unique inertial-based wearable assistive interface, embedded in a familiar head-worn garment, for individuals with severe motor impairmen…
Haptic bilateral teleoperation system for free-hand dental procedures
Department of Excellence 2023/2027 Project（卓越部门2023/2027项目）；Department of Information Engineering, Electrical Engineering and Applied Mathematics (DIEM), University of Salerno（信息工程、电气工程和应用数学系（DIEM），萨勒诺大学）；Specialization School in Oral Surgery, University of Naples Federico II（口腔外科专业学校，那不勒斯费德里科二世大学）
Model Predictive Control (MPC) schemes have proven their efficiency in controlling high degree-of-freedom (DoF) complex robotic systems. However, they come at a high computational cost and an update rate of about tens of hertz. This relatively slow update rate hinders the possibility of stable haptic teleoperation of such systems since the slow feedback loops can cause instabilities and loss of transparency to the operator. This work presents a novel framework for transparent teleoperation of MPC-controlled complex robotic systems. In particular, we employ a feedback MPC approach and exploit i…
Wrist-worn haptic interfaces can deliver a wide range of tactile cues for communication of information and interaction with virtual objects. Unlike fingertips, the wrist and forearm provide a considerably large area of skin that allows the placement of multiple haptic actuators as a display for enriching tactile information transfer with minimal encumbrance. Existing multi-degree-of-freedom (DoF) wrist-worn devices employ traditional rigid robotic mechanisms and electric motors that limit their versatility, miniaturization, distribution, and assembly. Alternative solutions based on soft elasto…
Nagoya University（名古屋大学）；Technical University of Darmstadt（达姆施塔特工业大学）
Vision-based tactile sensors have gained extensive attention in the robotics community. The sensors are highly expected to be capable of extracting contact information i.e. haptic information during in-hand manipulation. This nature of tactile sensors makes them a perfect match for haptic feedback applications. In this paper, we propose a contact force estimation method using the vision-based tactile sensor DIGIT, and apply it to a position-force teleoperation architecture for force feedback. The force estimation is done by building a depth map for DIGIT gel surface deformation measurement and…
Robotics System Lab (RSL), ETH Zurich（苏黎世联邦理工学院机器人系统实验室）；Dynamic Legged Systems Lab, Istituto Italiano di Tecnologia (IIT)（意大利理工学院动态腿足系统实验室）；Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente（特文特大学电气工程、数学与计算机科学学院）
When dealing with the haptic teleoperation of multi-limbed mobile manipulators, the problem of mitigating the destabilizing effects arising from the communication link between the haptic device and the remote robot has not been properly addressed. In this work, we propose a passive control architecture to haptically teleoperate a legged mobile manipulator, while remaining stable in the presence of time delays and frequency mismatches in the master and slave controllers. At the master side, a discrete-time energy modulation of the control input is proposed. At the slave side, passivity constrai…
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness and thus create full immersion, as well as offering the operator suitable control interfaces to achieve efficient and robust task fulfillment. We present a bimanual telemanipulation system consisting…
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness and thus create full immersion, as well as offering the operator suitable control interfaces to achieve efficient and robust task fulfillment. We present a bimanual telemanipulation system consisting…
Stanford University（斯坦福大学）；Palo Alto Veterans Affairs Health Care System（帕洛阿尔托退伍军人事务医疗系统）；Intuitive Surgical, Inc.（直觉外科公司）；Ben-Gurion University of the Negev（内盖夫本-古里安大学）
The lack of haptic feedback in Robot-assisted Minimally Invasive Surgery (RMIS) is a potential barrier to safe tissue handling during surgery. Bayesian modeling theory suggests that surgeons with experience in open or laparoscopic surgery can develop priors of tissue stiffness that translate to better force estimation abilities during RMIS compared to surgeons with no experience. To test if prior haptic experience leads to improved force estimation ability in teleoperation, 33 participants were assigned to one of three training conditions: manual manipulation, teleoperation with force feedback…
Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation
Queen Mary University of London（伦敦玛丽女王大学）；School of Engineering and Materials Science（工程与材料科学学院）；Humanoid AI（人形人工智能公司）；King’s College London（伦敦国王学院）；School of Biomedical Engineering and Imaging Sciences（生物医学工程与影像科学学院）
Institute of Perception, Action and Behaviour, School of Informatics, The University of Edinburgh（爱丁堡大学信息学院感知、行动与行为研究所）；Oxford Robotics Institute at the University of Oxford（牛津大学牛津机器人研究所）；School of Computer Science/Department of Electrical and Computer Engineering at McGill University（麦吉尔大学计算机科学学院/电气与计算机工程系）
This paper presents a control framework to teleoperate a quadruped robot's foot for operator-guided haptic exploration of the environment. Since one leg of a quadruped robot typically only has 3 actuated degrees of freedom (DoFs), the torso is employed to assist foot posture control via a hierarchical whole-body controller. The foot and torso postures are controlled by two analytical Cartesian impedance controllers cascaded by a null space projector. The contact forces acting on supporting feet are optimized by quadratic programming (QP). The foot's Cartesian impedance controller may also esti…
Research in multi-modal interfaces aims to provide solutions to immersion and increase overall human performance. A promising direction is combining auditory, visual and haptic interaction between the user and the simulated environment. However, no extensive comparisons exist to show how combining audiovisuohaptic interfaces affects human perception reflected on task performance. Our paper explores this idea. We present a thorough, full-factorial comparison of how all combinations of audio, visual and haptic interfaces affect performance during manipulation. We evaluate how each interface comb…
