During retinal microsurgery, precise manipulation of the delicate retinal tissue is required for positive surgical outcome. However, accurate manipulation and navigation of surgical tools remain difficult due to a constrained workspace and the top-down view during the surgery, which limits the surgeon's ability to estimate depth. To alleviate such difficulty, we propose to automate the tool-navigation task by learning to predict relative goal position on the retinal surface from the current tool-tip position. Given an estimated target on the retina, we generate an optimal trajectory leading to the predicted goal while imposing safety-related physical constraints aimed to minimize tissue damage. As an extended task, we generate goal predictions to various points across the retina to localize eye geometry and further generate safe trajectories within the estimated confines. Through experiments in both simulation and with several eye phantoms, we demonstrate that our framework can permit navigation to various points on the retina within 0.089mm and 0.118mm in xy error which is less than the human's surgeon mean tremor at the tool-tip of 0.180mm. All safety constraints were fulfilled and the algorithm was robust to previously unseen eyes as well as unseen objects in the scene. Live video demonstration is available here: https://youtu.be/n5j5jCCelXk
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）
Automatic laparoscope motion control is fundamentally important for surgeons to efficiently perform operations. However, its traditional control methods based on tool tracking without considering information hidden in surgical scenes are not intelligent enough, while the latest supervised imitation learning (IL)-based methods require expensive sensor data and suffer from distribution mismatch issues caused by limited demonstrations. In this paper, we propose a novel Imitation Learning framework for Laparoscope Control (ILLC) with reinforcement learning (RL), which can efficiently learn the con…
The Hong Kong University of Science and Technology（香港科技大学）
Vision-based autonomous driving through imitation learning mimics the behaviors of human drivers by training on pairs of data of raw driver-view images and actions. However, there are other cues, e.g. gaze behavior, available from human drivers that have yet to be exploited. Previous research has shown that novice human learners can benefit from observing experts' gaze patterns. We show here that deep neural networks can also benefit from this. We demonstrate different approaches to integrating gaze information into imitation networks. Our results show that the integration of gaze information …
RCM-ACT: Imitation Learning with Dynamic RCM Calibration for Autonomous Intraocular Foreign Body Removal
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；State Key Laboratory of Transvascular Implantation Devices of the Second Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院第二附属医院血管植入设备国家重点实验室）；Dessight Biomedical（Dessight生物医学公司）；Center for Rehabilitation Medicine, Department of Ophthalmology, Zhejiang Provincial People’s Hospital（浙江省人民医院康复医学中心、眼科部门）；School of Biosystems Engineering and Food Science, Zhejiang University（浙江大学生物系统工程与食品科学学院）；School of Public Health and Second Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院公共卫生学院及第二附属医院）；State Key Laboratory of Transvascular Implantation Devices of the Second Affiliated Hospital and School of Public Health, Zhejiang University School of Medicine（浙江大学医学院第二附属医院及公共卫生学院血管植入设备国家重点实验室）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）
The Quiet Eye Phenomenon in Minimally Invasive Surgery
Stanford University（斯坦福大学）；University of British Columbia（不列颠哥伦比亚大学）；Intuitive Surgical（直觉外科公司）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
Imitation Learning (IL) is an effective framework to learn visuomotor skills from offline demonstration data. However, IL methods often fail to generalize to new scene configurations not covered by training data. On the other hand, humans can manipulate objects in varying conditions. Key to such capability is hand-eye coordination, a cognitive ability that enables humans to adaptively direct their movements at task-relevant objects and be invariant to the objects' absolute spatial location. In this work, we present a learnable action space, Hand-eye Action Networks (HAN), that can approximate …
Surgeon hand tremor limits human capability during microsurgical procedures such as those that treat the eye. In contrast, elimination of hand tremor through the introduction of microsurgical robots diminishes the surgeon's tactile perception of useful and familiar tool-to-sclera forces. While the large mass and inertia of eye surgical robot prevents surgeon microtremor, loss of perception of small scleral forces may put the sclera at risk of injury. In this paper, we have applied and compared two different methods to assure the safety of sclera tissue during robot-assisted eye surgery. In the…
In this paper, a method to mimic a human face and eyes is proposed which can be regarded as a combination of computer vision techniques and neural network concepts. From a mechanical standpoint, a 3-DOF spherical parallel robot is used which imitates the human head movement. In what concerns eye movement, a 2-DOF mechanism is attached to the end-effector of the 3-DOF spherical parallel mechanism. In order to have robust and reliable results for the imitation, meaningful information should be extracted from the face mesh for obtaining the pose of a face, i.e., the roll, yaw, and pitch angles. T…
WM-DAgger: Enabling Efficient Data Aggregation for Imitation Learning with World Models
Peking University（北京大学）；JD Logistics（京东物流）；Nankai University（南开大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Rutgers University（罗格斯大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；Institut Polytechnique de Paris（巴黎综合理工学院）
While imitation learning provides us with an efficient toolkit to train robots, learning skills that are robust to environment variations remains a significant challenge. Current approaches address this challenge by relying either on large amounts of demonstrations that span environment variations or on handcrafted reward functions that require state estimates. Both directions are not scalable to fast imitation. In this work, we present Fast Imitation of Skills from Humans (FISH), a new imitation learning approach that can learn robust visual skills with less than a minute of human demonstrati…
Toward Autonomous Soft Robotic Endovascular Navigation via Imitation Learning
Johns Hopkins University（约翰霍普金斯大学）；Stanford University（斯坦福大学）；McGill University（麦吉尔大学）；University of Maryland（马里兰大学）；Swiss Federal Institute of Technology in Lausanne (EPFL)（日内瓦联邦理工学院（EPFL））；ETH Zurich（苏黎世联邦理工学院）
An accurate model of the environment and the dynamic agents acting in it offers great potential for improving motion planning. We present MILE: a Model-based Imitation LEarning approach to jointly learn a model of the world and a policy for autonomous driving. Our method leverages 3D geometry as an inductive bias and learns a highly compact latent space directly from high-resolution videos of expert demonstrations. Our model is trained on an offline corpus of urban driving data, without any online interaction with the environment. MILE improves upon prior state-of-the-art by 31% in driving sco…
Pre-Surgical Planner for Robot-Assisted Vitreoretinal Surgery: Integrating Eye Posture, Robot Position and Insertion Point
NSK Ltd.（恩斯克有限公司）；Technische Universität München（慕尼黑工业大学）；Klinikum rechts der Isar der Technische Universität München（慕尼黑工业大学右岸医院）；University of Alberta（阿尔伯塔大学）
University of California, Berkeley（加利福尼亚大学伯克利分校）；Honda Research Institute（本田研究所）
Sequential pulling policies to flatten and smooth fabrics have applications from surgery to manufacturing to home tasks such as bed making and folding clothes. Due to the complexity of fabric states and dynamics, we apply deep imitation learning to learn policies that, given color (RGB), depth (D), or combined color-depth (RGBD) images of a rectangular fabric sample, estimate pick points and pull vectors to spread the fabric to maximize coverage. To generate data, we develop a fabric simulator and an algorithmic supervisor that has access to complete state information. We train policies in sim…
Johns Hopkins University（约翰斯·霍普金斯大学）；Wilmer Eye Institute at the Johns Hopkins Hospital（约翰斯·霍普金斯医院威尔默眼科研究所）
A fundamental challenge in retinal surgery is safely navigating a surgical tool to a desired goal position on the retinal surface while avoiding damage to surrounding tissues, a procedure that typically requires tens-of-microns accuracy. In practice, the surgeon relies on depth-estimation skills to localize the tool-tip with respect to the retina in order to perform the tool-navigation task, which can be prone to human error. To alleviate such uncertainty, prior work has introduced ways to assist the surgeon by estimating the tool-tip distance to the retina and providing haptic or auditory fee…
Learning goal conditioned control in the real world is a challenging open problem in robotics. Reinforcement learning systems have the potential to learn autonomously via trial-and-error, but in practice the costs of manual reward design, ensuring safe exploration, and hyperparameter tuning are often enough to preclude real world deployment. Imitation learning approaches, on the other hand, offer a simple way to learn control in the real world, but typically require costly curated demonstration data and lack a mechanism for continuous improvement. Recently, iterative imitation techniques have …
Technische Universität München（慕尼黑工业大学）；Carl Zeiss Meditec（卡尔蔡司医疗技术公司）；Sun Yat-Sen University（中山大学）；Klinikum rechts der Isar der Technische Universität München（慕尼黑工业大学 Rechts der Isar 医院）
Ophthalmic microsurgery is known to be a challenging operation, which requires very precise and dexterous manipulation. Image guided robot-assisted surgery (RAS) is a promising solution that brings significant improvements in outcomes and reduces the physical limitations of human surgeons. However, this technology must be further developed before it can be routinely used in clinics. One of the problems is the lack of proper calibration between the robotic manipulator and appropriate imaging device. In this work, we developed a flexible framework for hand-eye calibration of an ophthalmic robot …
University of Verona（维罗纳大学）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
The current research focus in Robot-Assisted Minimally Invasive Surgery (RAMIS) is directed towards increasing the level of robot autonomy, to place surgeons in a supervisory position. Although Learning from Demonstrations (LfD) approaches are among the preferred ways for an autonomous surgical system to learn expert gestures, they require a high number of demonstrations and show poor generalization to the variable conditions of the surgical environment. In this work, we propose an LfD methodology based on Generative Adversarial Imitation Learning (GAIL) that is built on a Deep Reinforcement L…
SurgCalib: Gaussian Splatting-Based Hand-Eye Calibration for Robot-Assisted Minimally Invasive Surgery
Robotics and Control Laboratory, University of British Columbia（机器人与控制实验室，不列颠哥伦比亚大学）；Department of Biomedical Engineering, National University of Singapore（生物医学工程系，新加坡国立大学）；Division of Otolaryngology – Head and Neck Surgery, University of British Columbia（耳鼻喉科–头颈外科部，不列颠哥伦比亚大学）
Johns Hopkins University（约翰斯·霍普金斯大学）；Technische Universität München（慕尼黑工业大学）；The Chinese University of Hong Kong（香港中文大学）
Retinal surgery is a complex activity that can be challenging for a surgeon to perform effectively and safely. Image guided robot-assisted surgery is one of the promising solutions that bring significant surgical enhancement in treatment outcome and reduce the physical limitations of human surgeons. In this paper, we demonstrate a novel method for 3D guidance of the instrument based on the projection of spotlight in the single microscope images. The spotlight projection mechanism is firstly analyzed and modeled with a projection on both a plane and a sphere surface. To test the feasibility of …
One fundamental difficulty in robotic learning is the sim-real gap problem. In this work, we propose to use segmentation as the interface between perception and control, as a domain-invariant state representation. We identify two sources of sim-real gap, one is dynamics sim-real gap, the other is visual sim-real gap. To close dynamics sim-real gap, we propose to use closed-loop control. For complex task with segmentation mask input, we further propose to learn a closed-loop model-free control policy with deep neural network using imitation learning. To close visual sim-real gap, we propose to …
University of California, Berkeley（加州大学伯克利分校）；UC San Francisco East Bay（加州大学旧金山分校东湾）；SRI International（斯坦福国际研究院）
Automation of surgical tasks using cable-driven robots is challenging due to backlash, hysteresis, and cable tension, and these issues are exacerbated as surgical instruments must often be changed during an operation. In this work, we propose a framework for automation of high-precision surgical tasks by learning sample efficient, accurate, closed-loop policies that operate directly on visual feedback instead of robot encoder estimates. This framework, which we call intermittent visual servoing (IVS), intermittently switches to a learned visual servo policy for high-precision segments of repet…
Technische Universität München（慕尼黑工业大学）；Sun Yat-sen University（中山大学）；Johns Hopkins Hospital（约翰斯·霍普金斯医院）；Johns Hopkins University（约翰斯·霍普金斯大学）
Retinal surgery is a complex medical procedure that requires exceptional expertise and dexterity. For this purpose, several robotic platforms are currently being developed to enable or improve the outcome of microsurgical tasks. Since the control of such robots is often designed for navigation inside the eye in proximity to the retina, successful trocar docking and inserting the instrument into the eye represents an additional cognitive effort, and is, therefore, one of the open challenges in robotic retinal surgery. For this purpose, we present a platform for autonomous trocar docking that co…
Semi-Autonomous Laparoscopic Robot Docking with Learned Hand-Eye Information Fusion
School of Biomedical Engineering & Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与影像科学学院）；School of Mechatronical Engineering, Beijing Institute of Technology（北京理工大学机电学院）；School of Medical Technology, Beijing Institute of Technology（北京理工大学医学技术学院）
On-the-fly hand-eye calibration for the da Vinci surgical robot
Department of Mechanical Engineering, Imperial College London（帝国理工学院机械工程系）；Mechatronics in Medicine Laboratory（医学机电实验室）；Hamlyn Centre for Robotics Surgery（机器人外科哈姆林中心）
In the context of deep learning for robotics, we show effective method of training a real robot to grasp a tiny sphere (1.37cm of diameter), with an original combination of system design choices. We decompose the end-to-end system into a vision module and a closed-loop controller module. The two modules use target object segmentation as their common interface. The vision module extracts information from the robot end-effector camera, in the form of a binary segmentation mask of the target. We train it to achieve effective domain transfer by composing real background images with simulated image…
University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
How should we integrate representations from complementary sensors for autonomous driving? Geometry-based fusion has shown promise for perception (e.g. object detection, motion forecasting). However, in the context of end-to-end driving, we find that imitation learning based on existing sensor fusion methods underperforms in complex driving scenarios with a high density of dynamic agents. Therefore, we propose TransFuser, a mechanism to integrate image and LiDAR representations using self-attention. Our approach uses transformer modules at multiple resolutions to fuse perspective view and bird…
Control Modes of Teleoperated Surgical Robotic System's Tools in Ophthalmic Surgery
Department of Mechanical and Aerospace Engineering at the University of California, Los Angeles (UCLA)（加州大学洛杉矶分校机械与航空航天工程系）；Jules Stein Eye Institute at the University of California, Los Angeles (UCLA)（加州大学洛杉矶分校Jules Stein眼科研究所）
Computer vision and robotics are being increasingly applied in medical interventions. Especially in interventions where extreme precision is required they could make a difference. One such application is robot-assisted retinal microsurgery. In recent works, such interventions are conducted under a stereo-microscope, and with a robot-controlled surgical tool. The complementarity of computer vision and robotics has however not yet been fully exploited. In order to improve the robot control we are interested in 3D reconstruction of the anatomy and in automatic tool localization using a stereo mic…
