Autonomous surgery involves having surgical tasks performed by a robot operating under its own will, with partial or no human involvement. There are several important advantages of automation in surgery, which include increasing precision of care due to sub-millimeter robot control, real-time utilization of biosignals for interventional care, improvements to surgical efficiency and execution, and computer-aided guidance under various medical imaging and sensing modalities. While these methods may displace some tasks of surgical teams and individual surgeons, they also present new capabilities in interventions that are too difficult or go beyond the skills of a human. In this chapter, we provide an overview of robot autonomy in commercial use and in research, and present some of the challenges faced in developing autonomous surgical robots.
Robotic surgery has increased the domain of surgeries possible. Several examples of partial surgical automation have been seen in the past decade. We break down the path of automation tasks into features required and provide a checklist that can help reach higher levels of surgical automation. Finally, we discuss the current challenges and advances required to make this happen.
Johns Hopkins University（约翰斯·霍普金斯大学）；Children’s National Hospital（国立儿童医院）
Autonomous robotic surgery has the potential to provide efficacy, safety, and consistency independent of individual surgeons skill and experience. Autonomous soft-tissue surgery in unstructured and deformable environments is especially challenging as it necessitates intricate imaging, tissue tracking and surgical planning techniques, as well as a precise execution via highly adaptable control strategies. In the laparoscopic setting, soft-tissue surgery is even more challenging due to the need for high maneuverability and repeatability under motion and vision constraints. We demonstrate the fir…
bits-pilani（皮拉尼比特斯学院（注：此处为常见简称，完整应为Birla Institute of Technology and Science, Pilani，中文通常译为“比特拉理工学院”或简称“皮拉尼BITS”，但根据原文直接提取“bits-pilani”））
In this paper, we aim to achieve a human-robot work balance by implementing shared autonomy through a web interface. Shared autonomy integrates user input with the autonomous capabilities of the robot and therefore increases the overall performance of the robot. Presenting only the relevant information to the user on the web page lowers the cognitive load of the operator. Through our web interface, we provide a mechanism for the operator to directly interact using the displayed information by applying a point-and-click paradigm. Further, we present our idea to employ a human-robot mutual adapt…
Human operators in human-robot teams are commonly perceived to be critical for mission success. To explore the direct and perceived impact of operator input on task success and team performance, 16 real-world missions (10 hrs) were conducted based on the DARPA Subterranean Challenge. These missions were to deploy a heterogeneous team of robots for a search task to locate and identify artifacts such as climbing rope, drills and mannequins representing human survivors. Two conditions were evaluated: human operators that could control the robot team with state-of-the-art autonomy (Human-Robot Tea…
Northwestern University（西北大学）；Shirley Ryan AbilityLab（雪莉·瑞安能力实验室）；DePaul University（德保罗大学）
A dynamic autonomy allocation framework automatically shifts how much control lies with the human versus the robotics autonomy, for example based on factors such as environmental safety or user preference. To investigate the question of which factors should drive dynamic autonomy allocation, we perform a human subject study to collect ground truth data that shifts between levels of autonomy during shared-control robot operation. Information streams from the human, the interaction between the human and the robot, and the environment are analyzed. Machine learning methods -- both classical and d…
Carnegie Mellon University（卡内基梅隆大学）；National University of Singapore（新加坡国立大学）
Shared autonomy integrates user input with robot autonomy in order to control a robot and help the user to complete a task. Our work aims to improve the performance of such a human-robot team: the robot tries to guide the human towards an effective strategy, sometimes against the human's own preference, while still retaining his trust. We achieve this through a principled human-robot mutual adaptation formalism. We integrate a bounded-memory adaptation model of the human into a partially observable stochastic decision model, which enables the robot to adapt to an adaptable human. When the huma…
Trust-Preserved Human-Robot Shared Autonomy enabled by Bayesian Relational Event Modeling
School of Electrical and Computer Engineering, Georgia Institute of Technology（电子与计算机工程学院，佐治亚理工学院）；Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology（电子与计算机工程系，香港科技大学）
This paper presents a method for robot self-recognition and self-adaptation through the analysis of the contact between the robot end effector and its surrounding environment. Often, in off-line robot programming, the idealized robotic environment (the virtual one) does not reflect accurately the real one. In this situation, we are in the presence of a partially unknown environment (PUE). Thus, robotic systems must have some degree of autonomy to overcome this situation, especially when contact exists. The proposed force/motion control system has an external control loop based on forces and to…
Wheelchair-mounted robotic arms (and other assistive robots) should help their users perform everyday tasks. One way robots can provide this assistance is shared autonomy. Within shared autonomy, both the human and robot maintain control over the robot's motion: as the robot becomes confident it understands what the human wants, it increasingly intervenes to automate the task. But how does the robot know what tasks the human may want to perform in the first place? Today's shared autonomy approaches often rely on prior knowledge: for example, the robot must know the set of possible human goals …
Digital Surgery Limited（数字手术有限公司）；University College London（伦敦大学学院）；Kasetsart University（泰国农业大学）
Robotic surgery and novel surgical instrumentation present great potentials towards safer, more accurate and consistent minimally invasive surgery. However, their adoption is dependent to the access to training facilities and extensive surgical training. Robotic instruments require different dexterity skills compared to open or laparoscopic. Surgeons, therefore, are required to invest significant time by attending extensive training programs. Contrary, hands on experiences represent an additional operational cost for hospitals as the availability of robotic systems for training purposes is lim…
Long-term autonomy in service robotics is a current research topic, especially for dynamic, large-scale environments that change over time. We present Sobi, a mobile service robot developed as an interactive guide for open environments, such as public places with indoor and outdoor areas. The robot will serve as a platform for environmental modeling and human-robot interaction. Its main hardware and software components, which we freely license as a documented open source project, are presented. Another key focus is Sobi's monitoring system for long-term autonomy, which restores system componen…
The growth of scale and complexity of interactions between humans and robots highlights the need for new computational methods to automatically evaluate novel algorithms and applications. Exploring diverse scenarios of humans and robots interacting in simulation can improve understanding of the robotic system and avoid potentially costly failures in real-world settings. We formulate this problem as a quality diversity (QD) problem, where the goal is to discover diverse failure scenarios by simultaneously exploring both environments and human actions. We focus on the shared autonomy domain, whe…
When humans control robot arms these robots often need to infer the human's desired task. Prior research on assistive teleoperation and shared autonomy explores how robots can determine the desired task based on the human's joystick inputs. In order to perform this inference the robot relies on an internal mapping between joystick inputs and discrete tasks: e.g., pressing the joystick left indicates that the human wants a plate, while pressing the joystick right indicates a cup. This approach works well after the human understands how the robot interprets their inputs -- but inexperienced user…
Autonomy is the frontier of research in robotic surgery and its aim is to improve the quality of surgical procedures in the next future. One fundamental requirement for autonomy is advanced perception capability through vision sensors. In this paper, we propose a novel calibration technique for a surgical scenario with da Vinci robot. Calibration of the camera and the robot is necessary for precise positioning of the tools in order to emulate the high performance surgeons. Our calibration technique is tailored for RGB-D camera. Different tests performed on relevant use cases for surgery prove …
High levels of robot autonomy are a common goal, but there is a significant risk that the greater the autonomy of the robot the lesser the autonomy of the human working with the robot. For vulnerable populations like older adults who already have a diminished level of autonomy, this is an even greater concern. We propose that human autonomy needs to be at the center of the design for socially assistive robots. Towards this goal, we define autonomy and then provide architectural requirements for social robots to support the user's autonomy. As an example of a design effort, we describe some of …
Shantou University（汕头大学）；The First Affiliated Hospital of Medical College of Shantou University（汕头大学医学院第一附属医院）
This paper presents the development and experimental evaluation of a surgical robotic system for total hip arthroplasty (THA). Although existing robotic systems used in joint replacement surgery have achieved some progresses, the robot arm must be situated accurately at the target position during operation, which depends significantly on the experience of the surgeon. In addition, handheld acetabulum reamers typically exhibit uneven strength and grinding file. Moreover, the lack of techniques to real-time measure femoral neck length may lead to poor outcomes. To tackle these challenges, we pro…
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Kyung Hee University（庆熙大学）
Innovations from surgical robotic research rarely translates to live surgery due to the significant difference between the lab and a live environment. Live environments require considerations that are often overlooked during early stages of research such as surgical staff, surgical procedure, and the challenges of working with live tissue. One such example is the da Vinci Research Kit (dVRK) which is used by over 40 robotics research groups and represents an open-sourced version of the da Vinci Surgical System. Despite dVRK being available for nearly a decade and the ideal candidate for transl…
Assistive robots enable people with disabilities to conduct everyday tasks on their own. However, these tasks can be complex, containing both coarse reaching motions and fine-grained manipulation. For example, when eating, not only does one need to move to the correct food item, but they must also precisely manipulate the food in different ways (e.g., cutting, stabbing, scooping). Shared autonomy methods make robot teleoperation safer and more precise by arbitrating user inputs with robot controls. However, these works have focused mainly on the high-level task of reaching a goal from a discre…
Primary Experimental Feedback on a Co-manipulated Robotic System for Assisted Cervical Surgery
Pprime Institute（Pprime研究所）；University of Poitiers（普瓦捷大学）；CNRS（法国国家科学研究中心）；ISAE-ENSMA（法国高等航空航天学院-国立高等机械与航空技术学院）；Centre Hospitalier Universitaire (CHU)（大学教学医院（CHU））；National Central University（国立中央大学）；National Engineering School of Sousse（苏塞国立工程师学院）；University of Sousse（苏塞大学）
We describe a software framework and a hardware platform used in tandem for the design and analysis of robot autonomy algorithms in simulation and reality. The software, which is open source, containerized, and operating system (OS) independent, has three main components: a ROS 2 interface to a C++ vehicle simulation framework (Chrono), which provides high-fidelity wheeled/tracked vehicle and sensor simulation; a basic ROS 2-based autonomy stack for algorithm design and testing; and, a development ecosystem which enables visualization, and hardware-in-the-loop experimentation in perception, st…
Autonomous robotic surgery requires deliberation, i.e. the ability to plan and execute a task adapting to uncertain and dynamic environments. Uncertainty in the surgical domain is mainly related to the partial pre-operative knowledge about patient-specific anatomical properties. In this paper, we introduce a logic-based framework for surgical tasks with deliberative functions of monitoring and learning. The DEliberative Framework for Robot-Assisted Surgery (DEFRAS) estimates a pre-operative patient-specific plan, and executes it while continuously measuring the applied force obtained from a bi…
National University of Singapore（新加坡国立大学）；NUSRI Suzhou China（新加坡国立大学苏州研究院）；The Chinese University of Hong Kong（香港中文大学）；Imperial College London（伦敦帝国理工学院）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）
Generating a surgical report in robot-assisted surgery, in the form of natural language expression of surgical scene understanding, can play a significant role in document entry tasks, surgical training, and post-operative analysis. Despite the state-of-the-art accuracy of the deep learning algorithm, the deployment performance often drops when applied to the Target Domain (TD) data. For this purpose, we develop a multi-layer transformer-based model with the gradient reversal adversarial learning to generate a caption for the multi-domain surgical images that can describe the semantic relation…
Australian Centre for Field Robotics（澳大利亚现场机器人中心）；University of Sydney（悉尼大学）
Achieving long-term autonomy for mobile robots operating in real-world unstructured environments such as farms remains a significant challenge. This is made increasingly complex in the presence of moving humans or livestock. These environments require a robot to be adaptive in its immediate plans, accounting for the state of nearby individuals and the response that they might have to the robot's actions. Additionally, in order to achieve longer-term goals, consideration of the limited on-board resources available to the robot is required, especially for extended missions such as weeding an agr…
The University of Tokyo（东京大学）；Kitasato University Hospital（北里大学医院）
The limited workspace in pediatric endoscopic surgery makes surgical suturing one of the most difficult tasks. During suturing, surgeons have to prevent collisions between tools and also collisions with the surrounding tissues. Surgical robots have been shown to be effective in adult laparoscopy, but assistance for suturing in constrained workspaces has not been yet fully explored. In this letter, we propose guidance virtual fixtures to enhance the performance and the safety of suturing while generating the required task constraints using constrained optimization and Cartesian force feedback. …
MOSAIC: Modular Scalable Autonomy for Intelligent Coordination of Heterogeneous Robotic Teams
FZI Forschungszentrum für Informatik, Karlsruhe, Germany（弗赖堡信息研究中心，卡尔斯鲁厄，德国）；Robotic Systems Lab (RSL), ETH Zürich, Zürich, Switzerland（机器人系统实验室（RSL），苏黎世联邦理工学院，苏黎世，瑞士）；Machine Intelligence and Robotics Lab (MaiRo), Karlsruhe Institute for Technology (KIT), Karlsruhe, Germany（机器智能与机器人实验室（MaiRo），卡尔斯鲁厄技术学院（KIT），卡尔斯鲁厄，德国）；European Space Agency (ESA), Noordwijk, Netherlands（欧洲航天局（ESA），诺德威克，荷兰）
The control architecture of a spherical robot for Minimally Invasive Surgery
CESTER, Research Center for Industrial Robots Simulation and Testing, Technical University of Cluj-Napoca（克卢日-纳波卡技术大学工业机器人仿真与测试研究中心）；Technical Sciences Academy of Romania（罗马尼亚技术科学院）；École Centrale Nantes（南特中央理工大学）；Nantes Université（南特大学）；CNRS（法国国家科学研究中心）；LS2N, UMR 6004（南特数字科学实验室）
The surgical paradigm of Minimally Invasive Surgery (MIS) has been a key driver to the adoption of robotic surgical assistance. Progress in the last three decades has led to a gradual transition from manual laparoscopic surgery with rigid instruments to robot-assisted surgery. In the last decade, the increasing demand for new surgical paradigms to enable access into the anatomy without skin incision (intraluminal surgery) or with a single skin incision (Single Port Access surgery - SPA) has led researchers to investigate snake-like flexible surgical devices. In this chapter, we first present a…
University of California San Diego（加州大学圣地亚哥分校）；University of New Mexico（新墨西哥大学）
Automating the process of manipulating and delivering sutures during robotic surgery is a prominent problem at the frontier of surgical robotics, as automating this task can significantly reduce surgeons' fatigue during tele-operated surgery and allow them to spend more time addressing higher-level clinical decision making. Accomplishing autonomous suturing and suture manipulation in the real world requires accurate suture thread localization and reconstruction, the process of creating a 3D shape representation of suture thread from 2D stereo camera surgical image pairs. This is a very challen…
In shared autonomy, user input and robot autonomy are combined to control a robot to achieve a goal. Often, the robot does not know a priori which goal the user wants to achieve, and must both predict the user's intended goal, and assist in achieving that goal. We formulate the problem of shared autonomy as a Partially Observable Markov Decision Process with uncertainty over the user's goal. We utilize maximum entropy inverse optimal control to estimate a distribution over the user's goal based on the history of inputs. Ideally, the robot assists the user by solving for an action which minimiz…
Design Analysis of an Innovative Parallel Robot for Minimally Invasive Pancreatic Surgery
Technical University of Cluj-Napoca（克卢日-纳波卡技术大学）；Technical Sciences Academy of Romania（罗马尼亚技术科学院）；École Centrale Nantes（南特中央理工学校）；Nantes Université（南特大学）；CNRS（法国国家科学研究中心）
Institute for Perception, Action and Behaviour, School of Informatics, The University of Edinburgh（爱丁堡大学信息学院感知、行动与行为研究所）
The deployment of robots in industrial and civil scenarios is a viable solution to protect operators from danger and hazards. Shared autonomy is paramount to enable remote control of complex systems such as legged robots, allowing the operator to focus on the essential tasks instead of overly detailed execution. To realize this, we propose a comprehensive control framework for inspection and intervention using a legged robot and validate the integration of multiple loco-manipulation algorithms optimised for improving the remote operation. The proposed control offers 3 operation modes: fully au…
ENSTA-ParisTech（巴黎高等先进技术学院）；Brown University（布朗大学）；University of Georgia（佐治亚大学）；University of Twente（特文特大学）；Vrije Universiteit Brussel（布鲁塞尔自由大学）；University of Plymouth（普利茅斯大学）；UC San Diego（加州大学圣地亚哥分校）；University of Lincoln（林肯大学）；Mississippi State University（密西西比州立大学）
Robot-Assisted Therapy (RAT) has successfully been used in HRI research by including social robots in health-care interventions by virtue of their ability to engage human users both social and emotional dimensions. Research projects on this topic exist all over the globe in the USA, Europe, and Asia. All of these projects have the overall ambitious goal to increase the well-being of a vulnerable population. Typical work in RAT is performed using remote controlled robots; a technique called Wizard-of-Oz (WoZ). The robot is usually controlled, unbeknownst to the patient, by a human operator. How…
LIRMM, University of Montpellier, CNRS（蒙彼利埃大学法国国家科学研究中心LIRMM实验室）；Stanford University（斯坦福大学）
The current generation of surgeons requires extensive training in teleoperation to develop specific dexterous skills, which are independent of medical knowledge. Training curricula progress from manipulation tasks to simulated surgical tasks but are limited in time. To tackle this, we propose to integrate surgical robotic training together with Haptic Feedback (HF) to improve skill acquisition. This paper present the initial but promising results of our haptic device designed to support in the training of surgical gestures. Our ongoing work is related to integrate the HF in the RAVEN II platfo…
University of Verona（维罗纳大学）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
Deep Reinforcement Learning (DRL) is a viable solution for automating repetitive surgical subtasks due to its ability to learn complex behaviours in a dynamic environment. This task automation could lead to reduced surgeon's cognitive workload, increased precision in critical aspects of the surgery, and fewer patient-related complications. However, current DRL methods do not guarantee any safety criteria as they maximise cumulative rewards without considering the risks associated with the actions performed. Due to this limitation, the application of DRL in the safety-critical paradigm of robot…
Ben-Gurion University of the Negev（内盖夫本-古里安大学）；California Institute of Technology（加州理工学院）；University of Southern California（南加州大学）；Soroka Medical Center（索罗卡医疗中心）
Teleoperated robot-assisted minimally-invasive surgery (RAMIS) offers many advantages over open surgery. However, there are still no guidelines for training skills in RAMIS. Motor learning theories have the potential to improve the design of RAMIS training but they are based on simple movements that do not resemble the complex movements required in surgery. To fill this gap, we designed an experiment to investigate the effect of time-dependent force perturbations on the learning of a pattern-cutting surgical task. Thirty participants took part in the experiment: (1) a control group that traine…
