Force estimation using neural networks is a promising approach to enable haptic feedback in minimally invasive surgical robots without end-effector force sensors. Various network architectures have been proposed, but none have been tested in real time with surgical-like manipulations. Thus, questions remain about the real-time transparency and stability of force feedback from neural network-based force estimates. We characterize the real-time impedance transparency and stability of force feedback rendered on a da Vinci Research Kit teleoperated surgical robot using neural networks with vision-only, state-only, and state and vision inputs. Networks were trained on an existing dataset of teleoperated manipulations without force feedback. To measure real-time stability and transparency during teleoperation with force feedback to the operator, we modeled a one-degree-of-freedom human and surgeon-side manipulandum that moved the patient-side robot to perform manipulations on silicone artificial tissue over various robot and camera configurations, and tools. We found that the networks using state inputs displayed more transparent impedance than a vision-only network. However, state-based networks displayed large instability when used to provide force feedback during lateral manipulation of the silicone. In contrast, the vision-only network showed consistent stability in all the evaluated directions. We confirmed the performance of the vision-only network for real-time force feedback in a demonstration with a human teleoperator.
This paper presents a novel approach to haptic teleoperation. Specifically, we use control barrier functions (CBFs) to generate force feedback to help human operators safely fly quadrotor UAVs. CBFs take a control signal as input and output a control signal that is as close as possible to the initial control signal, while also meeting specified safety constraints. In the proposed method, we generate haptic force feedback based on the difference between a command issued by the human operator and the safe command returned by a CBF. In this way, if the user issues an unsafe control command, the h…
We present a novel haptic teleoperation approach that considers not only the safety but also the stability of a teleoperation system. Specifically, we build upon previous work on haptic shared control, which uses control barrier functions (CBFs) to generate a reference haptic feedback that informs the human operator on the internal state of the system, helping them to safely navigate the robot without taking away their control authority. Crucially, in this approach the force rendered to the user is not directly reflected in the motion of the robot (which is still directly controlled by the use…
University of Sussex（萨塞克斯大学）；University of Edinburgh（爱丁堡大学）
The teleoperation of robots enables remote intervention in distant and dangerous tasks without putting the operator in harm's way. However, remote operation faces fundamental challenges due to limits in communication delays. The proposed work improves the performances of teleoperation architecture based on Fractal Impedance Controller (FIC) by integrating into the haptic teleoperation pipeline a postural optimisation that also accounts for the replica robots' physical limitations. This update improves dynamic interactions by trading off tracking accuracy to maintain the system within its power…
Idiap Research Institute（伊迪亚普研究所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Haptic guidance is a powerful technique to combine the strengths of humans and autonomous systems for teleoperation. The autonomous system can provide haptic cues to enable the operator to perform precise movements; the operator can interfere with the plan of the autonomous system leveraging his/her superior cognitive capabilities. However, providing haptic cues such that the individual strengths are not impaired is challenging because low forces provide little guidance, whereas strong forces can hinder the operator in realizing his/her plan. Based on variational inference, we learn a Gaussian…
University of the West of England（西英格兰大学）；Toshiba Europe Ltd.（东芝欧洲有限公司）
Haptic teleoperation is typically realized through wired networking technologies (e.g., Ethernet) which guarantee performance of control loops closed over the communication medium, particularly in terms of latency, jitter, and reliability. This demonstration shows the capability of conducting haptic teleoperation over a novel low-power wireless control technology, called GALLOP, in a nuclear decommissioning use-case. It shows the viability of GALLOP for meeting latency, timeliness, and safety requirements of haptic teleoperation. Evaluation conducted as part of the demonstration reveals that G…
Shared autonomy teleoperation can guarantee safety, but does so by reducing the human operator's control authority, which can lead to reduced levels of human-robot agreement and user satisfaction. This paper presents a novel haptic shared autonomy teleoperation paradigm that uses haptic feedback to inform the user about the inner state of a shared autonomy paradigm, while still guaranteeing safety. This differs from haptic shared control, which uses haptic feedback to inform the user's actions, but gives the human operator full control over the robot's actions. We conducted a user study in whi…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Drone teleoperation is usually accomplished using remote radio controllers, devices that can be hard to master for inexperienced users. Moreover, the limited amount of information fed back to the user about the robot's state, often limited to vision, can represent a bottleneck for operation in several conditions. In this work, we present a wearable interface for drone teleoperation and its evaluation through a user study. The two main features of the proposed system are a data glove to allow the user to control the drone trajectory by hand motion and a haptic system used to augment their aware…
Safety-Critical Bilateral Teleoperation for Omnidirectional Aerial Manipulation Using Force-Sensorless Haptic Feedback
Seoul National University（首尔大学）；Automation and System Research Institute (ASRI)（自动化与系统研究所）；Institute of Advanced Aerospace Technology (IAAT)（先进航空航天技术研究所）
Haptic bilateral teleoperation system for free-hand dental procedures
Department of Excellence 2023/2027 Project（卓越部门2023/2027项目）；Department of Information Engineering, Electrical Engineering and Applied Mathematics (DIEM), University of Salerno（信息工程、电气工程和应用数学系（DIEM），萨勒诺大学）；Specialization School in Oral Surgery, University of Naples Federico II（口腔外科专业学校，那不勒斯费德里科二世大学）
Model Predictive Control (MPC) schemes have proven their efficiency in controlling high degree-of-freedom (DoF) complex robotic systems. However, they come at a high computational cost and an update rate of about tens of hertz. This relatively slow update rate hinders the possibility of stable haptic teleoperation of such systems since the slow feedback loops can cause instabilities and loss of transparency to the operator. This work presents a novel framework for transparent teleoperation of MPC-controlled complex robotic systems. In particular, we employ a feedback MPC approach and exploit i…
Nagoya University（名古屋大学）；Technical University of Darmstadt（达姆施塔特工业大学）
Vision-based tactile sensors have gained extensive attention in the robotics community. The sensors are highly expected to be capable of extracting contact information i.e. haptic information during in-hand manipulation. This nature of tactile sensors makes them a perfect match for haptic feedback applications. In this paper, we propose a contact force estimation method using the vision-based tactile sensor DIGIT, and apply it to a position-force teleoperation architecture for force feedback. The force estimation is done by building a depth map for DIGIT gel surface deformation measurement and…
Robotics System Lab (RSL), ETH Zurich（苏黎世联邦理工学院机器人系统实验室）；Dynamic Legged Systems Lab, Istituto Italiano di Tecnologia (IIT)（意大利理工学院动态腿足系统实验室）；Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente（特文特大学电气工程、数学与计算机科学学院）
When dealing with the haptic teleoperation of multi-limbed mobile manipulators, the problem of mitigating the destabilizing effects arising from the communication link between the haptic device and the remote robot has not been properly addressed. In this work, we propose a passive control architecture to haptically teleoperate a legged mobile manipulator, while remaining stable in the presence of time delays and frequency mismatches in the master and slave controllers. At the master side, a discrete-time energy modulation of the control input is proposed. At the slave side, passivity constrai…
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness and thus create full immersion, as well as offering the operator suitable control interfaces to achieve efficient and robust task fulfillment. We present a bimanual telemanipulation system consisting…
Stanford University（斯坦福大学）；Palo Alto Veterans Affairs Health Care System（帕洛阿尔托退伍军人事务医疗系统）；Intuitive Surgical, Inc.（直觉外科公司）；Ben-Gurion University of the Negev（内盖夫本-古里安大学）
The lack of haptic feedback in Robot-assisted Minimally Invasive Surgery (RMIS) is a potential barrier to safe tissue handling during surgery. Bayesian modeling theory suggests that surgeons with experience in open or laparoscopic surgery can develop priors of tissue stiffness that translate to better force estimation abilities during RMIS compared to surgeons with no experience. To test if prior haptic experience leads to improved force estimation ability in teleoperation, 33 participants were assigned to one of three training conditions: manual manipulation, teleoperation with force feedback…
Robotic teleoperation is a key technology for a wide variety of applications. It allows sending robots instead of humans in remote, possibly dangerous locations while still using the human brain with its enormous knowledge and creativity, especially for solving unexpected problems. A main challenge in teleoperation consists of providing enough feedback to the human operator for situation awareness and thus create full immersion, as well as offering the operator suitable control interfaces to achieve efficient and robust task fulfillment. We present a bimanual telemanipulation system consisting…
Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation
Queen Mary University of London（伦敦玛丽女王大学）；School of Engineering and Materials Science（工程与材料科学学院）；Humanoid AI（人形人工智能公司）；King’s College London（伦敦国王学院）；School of Biomedical Engineering and Imaging Sciences（生物医学工程与影像科学学院）
Institute of Perception, Action and Behaviour, School of Informatics, The University of Edinburgh（爱丁堡大学信息学院感知、行动与行为研究所）；Oxford Robotics Institute at the University of Oxford（牛津大学牛津机器人研究所）；School of Computer Science/Department of Electrical and Computer Engineering at McGill University（麦吉尔大学计算机科学学院/电气与计算机工程系）
This paper presents a control framework to teleoperate a quadruped robot's foot for operator-guided haptic exploration of the environment. Since one leg of a quadruped robot typically only has 3 actuated degrees of freedom (DoFs), the torso is employed to assist foot posture control via a hierarchical whole-body controller. The foot and torso postures are controlled by two analytical Cartesian impedance controllers cascaded by a null space projector. The contact forces acting on supporting feet are optimized by quadratic programming (QP). The foot's Cartesian impedance controller may also esti…
University of Siena（锡耶纳大学）；Istituto Italiano di Tecnologia（意大利技术研究院）；University of Padua（帕多瓦大学）
A novel sensory substitution technique is presented. Kinesthetic and cutaneous force feedback are substituted by cutaneous feedback (CF) only, provided by two wearable devices able to apply forces to the index finger and the thumb, while holding a handle during a teleoperation task. The force pattern, fed back to the user while using the cutaneous devices, is similar, in terms of intensity and area of application, to the cutaneous force pattern applied to the finger pad while interacting with a haptic device providing both cutaneous and kinesthetic force feedback. The pattern generated using t…
Research in multi-modal interfaces aims to provide solutions to immersion and increase overall human performance. A promising direction is combining auditory, visual and haptic interaction between the user and the simulated environment. However, no extensive comparisons exist to show how combining audiovisuohaptic interfaces affects human perception reflected on task performance. Our paper explores this idea. We present a thorough, full-factorial comparison of how all combinations of audio, visual and haptic interfaces affect performance during manipulation. We evaluate how each interface comb…
CDF-Glove: A Cable-Driven Force Feedback Glove for Dexterous Teleoperation
Center for Artificial Intelligence and Robotics, Shenzhen International Graduate School, Tsinghua University（人工智能与机器人中心，深圳国际研究生院，清华大学）；PKU-Psibot Joint Lab（北京大学-PsiBot联合实验室）；Department of Advanced Manufacturing, Shenzhen International Graduate School, Tsinghua University（先进制造系，深圳国际研究生院，清华大学）
This paper develops a novel force observer for bilateral teleoperation systems. Type-2 fuzzy logic is used to describe the overall dynamic system, and Moving Horizon Estimation (MHE) is employed to assess clean states as well as the values of dynamic uncertainties, and simultaneously filter out the measurement noises, which guarantee the high degree of accuracy for the observed forces. Compared with the existing methods, the proposed force observer can run without knowing exact mathematical dynamic functions and is robust to different kinds of noises. A force-reflection four-channel teleoperat…
