3-D pose estimation of instruments is a crucial step towards automatic scene understanding in robotic minimally invasive surgery. Although robotic systems can potentially directly provide joint values, this information is not commonly exploited inside the operating room, due to its possible unreliability, limited access and the time-consuming calibration required, especially for continuum robots. For this reason, standard approaches for 3-D pose estimation involve the use of external tracking systems. Recently, image-based methods have emerged as promising, non-invasive alternatives. While many image-based approaches in the literature have shown accurate results, they generally require either a complex iterative optimization for each processed image, making them unsuitable for real-time applications, or a large number of manually-annotated images for efficient learning. In this paper we propose a self-supervised image-based method, exploiting, at training time only, the imprecise kinematic information provided by the robot. In order to avoid introducing time-consuming manual annotations, the problem is formulated as an auto-encoder, smartly bottlenecked by the presence of a physical model of the robotic instruments and surgical camera, forcing a separation between image background and kinematic content. Validation of the method was performed on semi-synthetic, phantom and in-vivo datasets, obtained using a flexible robotized endoscope, showing promising results for real-time image-based 3-D pose estimation of surgical instruments.
SurgicalPart-SAM: Part-to-Whole Collaborative Prompting for Surgical Instrument Segmentation
School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；School of Software Engineering, South China University of Technology（华南理工大学软件工程学院）；Department of Data Science & AI, Monash University（莫纳什大学数据科学与人工智能系）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机科学学院）；Department of Computer Science, University of Rochester（罗切斯特大学计算机科学系）
本文提出 SurgicalPart-SAM (SP-SAM)，通过协同提示、跨模态编码器和部分到整体自适应融合与分层解码，将器械结构知识与 SAM 集成，解决手术器械分割中结构复杂和提示不灵活的问题，以最少的可调参数实现了最先进的性能。
SurgTrack: CAD-Free 3D Tracking of Real-world Surgical Instruments
Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS（香港中文大学（深圳）人工智能与机器人中心（香港中文大学（深圳）与中国科学院深圳先进技术研究院联合机构））；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
SurgRIPE challenge: Benchmark of Surgical Robot Instrument Pose Estimation
The Hamlyn Centre for Robotic Surgery, Imperial College London, United Kingdom；ImFusion GmbH, Munich, Germany；Technical University of Munich, Munich, Germany；Eindhoven University of Technology, Eindhoven, Netherlands；Department of Engineering Physics, Tsinghua University, Beijing, China；Department of Transdisciplinary Medicine, Seoul National University Hospital, South Korea；School of Computing, University of Leeds, United Kingdom；STORMLab, University of Leeds, United Kingdom
Surgical robots are usually controlled using a priori models based on the robots' geometric parameters, which are calibrated before the surgical procedure. One of the challenges in using robots in real surgical settings is that those parameters can change over time, consequently deteriorating control accuracy. In this context, our group has been investigating online calibration strategies without added sensors. In one step toward that goal, we have developed an algorithm to estimate the pose of the instruments' shafts in endoscopic images. In this study, we build upon that earlier work and pro…
ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding
Centre for Artificial Intelligence and Robotics (CAIR), Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；Chinese University of Hong Kong（香港中文大学）
State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统管理与控制国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
The real-time segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, it is still a challenging task to implement deep learning models to do real-time segmentation for surgical instruments due to their high computational costs and slow inference speed. In this paper, we propose an attention-guided lightweight network (LWANet), which can segment surgical instruments in real-time. LWANet adopts encoder-decoder architecture, where the encoder is the lightweight network MobileNetV2, and the decoder consists of depthwise separable convolution, attention fusion …
SurgPose: Generalisable Surgical Instrument Pose Estimation using Zero-Shot Learning and Stereo Vision
Department of Computing, Imperial College London（帝国理工学院计算系）；Hamlyn Centre for Robotic Surgery, Department of Surgery and Cancer, Imperial College London（帝国理工学院外科与癌症部门机器人手术中心）
Surgical robots are used to perform minimally invasive surgery and alleviate much of the burden imposed on surgeons. Our group has developed a surgical robot to aid in the removal of tumors at the base of the skull via access through the nostrils. To avoid injuring the patients, a collision-avoidance algorithm that depends on having an accurate model for the poses of the instruments' shafts is used. Given that the model's parameters can change over time owing to interactions between instruments and other disturbances, the online estimation of the poses of the instrument's shaft is essential. I…
This work proves that semantic segmentation on minimally invasive surgical instruments can be improved by using training data that has been augmented through domain adaptation. The benefit of this method is twofold. Firstly, it suppresses the need of manually labeling thousands of images by transforming synthetic data into realistic-looking data. To achieve this, a CycleGAN model is used, which transforms a source dataset to approximate the domain distribution of a target dataset. Secondly, this newly generated data with perfect labels is utilized to train a semantic segmentation neural networ…
Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning
University of Macau（澳门大学）；University of Macau Advanced Research Institute in Hengqin（澳门大学横琴研究院）；The Chinese University of Hong Kong（香港中文大学）；The Hong Kong Polytechnic University（香港理工大学）；Shanghai Jiao Tong University（上海交通大学）；Tongren Hospital（同仁医院）；Duke University（杜克大学）
University of British Columbia（不列颠哥伦比亚大学）；Technical University of Munich（慕尼黑工业大学）
Surgical instrument segmentation for robot-assisted surgery is needed for accurate instrument tracking and augmented reality overlays. Therefore, the topic has been the subject of a number of recent papers in the CAI community. Deep learning-based methods have shown state-of-the-art performance for surgical instrument segmentation, but their results depend on labelled data. However, labelled surgical data is of limited availability and is a bottleneck in surgical translation of these methods. In this paper, we demonstrate the limited generalizability of these methods on different datasets, inc…
Chongqing University（重庆大学）；Chongqing University-University of Cincinnati Joint Co-op Institute（重庆大学-辛辛那提大学联合学院）；Imperial College London（帝国理工学院）
Miniaturized instruments are highly needed for robot assisted medical healthcare and treatment, especially for less invasive surgery as it empowers more flexible access to restricted anatomic intervention. But the robotic design is more challenging due to the contradictory needs of miniaturization and the capability of manipulating with a large dexterous workspace. Thus, kinematic parameter optimization is of great significance in this case. To this end, this paper proposes an approach based on dexterous workspace determination for designing a miniaturized tendon-driven surgical instrument und…
University of California, Berkeley（加州大学伯克利分校）；UC San Francisco East Bay（加州大学旧金山分校东湾）；SRI International（斯坦福国际研究院）
Automation of surgical tasks using cable-driven robots is challenging due to backlash, hysteresis, and cable tension, and these issues are exacerbated as surgical instruments must often be changed during an operation. In this work, we propose a framework for automation of high-precision surgical tasks by learning sample efficient, accurate, closed-loop policies that operate directly on visual feedback instead of robot encoder estimates. This framework, which we call intermittent visual servoing (IVS), intermittently switches to a learned visual servo policy for high-precision segments of repet…
Towards Autonomous Instrument Tray Assembly for Sterile Processing Applications
Department of Mechanical Engineering, University of Louisville（机械工程系，路易斯维尔大学）；Sterile Processing Department, Saint Vincent’s Hospital（无菌处理部，圣文森医院）；Department of Orthopedic Surgery, University of Louisville（骨科手术系，路易斯维尔大学）；Department of Neurological Surgery, University of Massachusetts（神经外科系，马萨诸塞大学）
Instrument-Splatting++: Towards Controllable Surgical Instrument Digital Twin Using Gaussian Splatting
Department of Biomedical Engineering, National University of Singapore (NUS)（生物医学工程系，新加坡国立大学）；Department of Electrical and Computer Engineering, The University of British Columbia（电气与计算机工程系，不列颠哥伦比亚大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Surgical instrument segmentation is extremely important for computer-assisted surgery. Different from common object segmentation, it is more challenging due to the large illumination and scale variation caused by the special surgical scenes. In this paper, we propose a novel bilinear attention network with adaptive receptive field to solve these two challenges. For the illumination variation, the bilinear attention module can capture second-order statistics to encode global contexts and semantic dependencies between local pixels. With them, semantic features in challenging areas can be inferre…
Design of an innovative robotic surgical instrument for circular stapling
University of Medicine and Pharmacy “Iuliu Hațieganu”（“尤利乌·哈蒂埃加努”医药大学）；École Centrale Nantes（南特中央理工学校）；Nantes Université（南特大学）；Technical Sciences Academy of Romania（罗马尼亚技术科学院）
Robotic-assisted surgeries benefit both surgeons and patients, however, surgeons frequently need to adjust the endoscopic camera to achieve good viewpoints. Simultaneously controlling the camera and the surgical instruments is impossible, and consequentially, these camera adjustments repeatedly interrupt the surgery. Autonomous camera control could help overcome this challenge, but most existing systems are reactive, e.g., by having the camera follow the surgical instruments. We propose a predictive approach for anticipating when camera movements will occur using artificial neural networks. We…
University of California San Diego（加州大学圣迭戈分校）；East China Normal University（华东师范大学）
Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and re…
Digital Surgery Limited（数字手术有限公司）；University College London（伦敦大学学院）；Kasetsart University（泰国农业大学）
Robotic surgery and novel surgical instrumentation present great potentials towards safer, more accurate and consistent minimally invasive surgery. However, their adoption is dependent to the access to training facilities and extensive surgical training. Robotic instruments require different dexterity skills compared to open or laparoscopic. Surgeons, therefore, are required to invest significant time by attending extensive training programs. Contrary, hands on experiences represent an additional operational cost for hospitals as the availability of robotic systems for training purposes is lim…
NUS Graduate School for Integrative Sciences and Engineering (NGS)（新加坡国立大学综合科学与工程研究生院（NGS））；National University of Singapore（新加坡国立大学）；University of Moratuwa（莫拉图瓦大学）；NIT Trichy（印度国立理工学院蒂鲁吉拉伯利分校）
Robot-assisted surgery is an emerging technology which has undergone rapid growth with the development of robotics and imaging systems. Innovations in vision, haptics and accurate movements of robot arms have enabled surgeons to perform precise minimally invasive surgeries. Real-time semantic segmentation of the robotic instruments and tissues is a crucial step in robot-assisted surgery. Accurate and efficient segmentation of the surgical scene not only aids in the identification and tracking of instruments but also provided contextual information about the different tissues and instruments be…
In recent years, surgical robots have become instrumental tools for assisting surgeons in performing complex surgical procedures in hospitals. Unlike conventional surgical methods, robotic systems help surgeons, for example, to perform minimally invasive surgical procedures while enhancing the precision and control of operations (e.g. tiny incisions, wound sutures, endoscopic suturing, among others). To this extent, it is essential to consider several factors that may influence the feasibility and decision making of employing robotic systems in surgical procedures. In this paper, we propose an…
Vitreoretinal surgery is challenging even for expert surgeons owing to the delicate target tissues and the diminutive workspace in the retina. In addition to improved dexterity and accuracy, robot assistance allows for (partial) task automation. In this work, we propose a strategy to automate the motion of the light guide with respect to the surgical instrument. This automation allows the instrument's shadow to always be inside the microscopic view, which is an important cue for the accurate positioning of the instrument in the retina. We show simulations and experiments demonstrating that the…
STRAS (Single access Transluminal Robotic Assistant for Surgeons) is a new robotic system for application to intraluminal surgical procedures. Preclinical testing of STRAS has recently permitted to demonstrate major advantages of the system in comparison with classic procedures. Benchmark methods permitting to establish objective criteria for expertise need to be worked out now to effectively train surgeons on this new system in the near future. STRAS consists of three cable driven subsystems, one endoscope serving as guide, and two flexible instruments. The flexible instruments have three deg…
High-Precision Surgical Robotic System for Intraocular Procedures
Department of Mechanical and Aerospace Engineering, University of California, Los Angeles（机械与航空航天工程系，加州大学洛杉矶分校）；Stein Eye Institute, University of California, Los Angeles（斯坦眼科研究所，加州大学洛杉矶分校）
Politecnico di Milano（米兰理工大学）；University College London（伦敦大学学院）
Robot-assisted laparoscopic prostatectomy (RALP) is a treatment for prostate cancer that involves complete or nerve sparing removal prostate tissue that contains cancer. After removal the bladder neck is successively sutured directly with the urethra. The procedure is called urethrovesical anastomosis and is one of the most dexterity demanding tasks during RALP. Two suturing instruments and a pair of needles are used in combination to perform a running stitch during urethrovesical anastomosis. While robotic instruments provide enhanced dexterity to perform the anastomosis, it is still highly c…
KU Leuven（鲁汶大学）；University College London（伦敦大学学院）；King’s College London（伦敦国王学院）
Many keyhole interventions rely on bi-manual handling of surgical instruments, forcing the main surgeon to rely on a second surgeon to act as a camera assistant. In addition to the burden of excessively involving surgical staff, this may lead to reduced image stability, increased task completion time and sometimes errors due to the monotony of the task. Robotic endoscope holders, controlled by a set of basic instructions, have been proposed as an alternative, but their unnatural handling may increase the cognitive load of the (solo) surgeon, which hinders their clinical acceptance. More seamle…
