Current laparoscopic camera motion automation relies on rule-based approaches or only focuses on surgical tools. Imitation Learning (IL) methods could alleviate these shortcomings, but have so far been applied to oversimplified setups. Instead of extracting actions from oversimplified setups, in this work we introduce a method that allows to extract a laparoscope holder's actions from videos of laparoscopic interventions. We synthetically add camera motion to a newly acquired dataset of camera motion free da Vinci surgery image sequences through a novel homography generation algorithm. The synthetic camera motion serves as a supervisory signal for camera motion estimation that is invariant to object and tool motion. We perform an extensive evaluation of state-of-the-art (SOTA) Deep Neural Networks (DNNs) across multiple compute regimes, finding our method transfers from our camera motion free da Vinci surgery dataset to videos of laparoscopic interventions, outperforming classical homography estimation approaches in both, precision by 41%, and runtime on a CPU by 43%.
University Hospital Heidelberg（海德堡大学医院）；Otto-von-Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）
Minimally Invasive Surgery (MIS) techniques have gained rapid popularity among surgeons since they offer significant clinical benefits including reduced recovery time and diminished post-operative adverse effects. However, conventional endoscopic systems output monocular video which compromises depth perception, spatial orientation and field of view. Suturing is one of the most complex tasks performed under these circumstances. Key components of this tasks are the interplay between needle holder and the surgical needle. Reliable 3D localization of needle and instruments in real time could be u…
Imperial College London（伦敦帝国理工学院）；University of Oxford（牛津大学）；Lightpoint Medical Ltd.（光点医疗有限公司）
Dense depth estimation and 3D reconstruction of a surgical scene are crucial steps in computer assisted surgery. Recent work has shown that depth estimation from a stereo images pair could be solved with convolutional neural networks. However, most recent depth estimation models were trained on datasets with per-pixel ground truth. Such data is especially rare for laparoscopic imaging, making it hard to apply supervised depth estimation to real surgical applications. To overcome this limitation, we propose SADepth, a new self-supervised depth estimation method based on Generative Adversarial N…
UGA（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；Univ Rennes（雷恩大学）；LTSI（图像与信号处理实验室）；Michallon University Hospital（米沙隆大学医院）；CHU Grenoble（格勒诺布尔大学医院）
Objective: A median of 14.4% of patient undergone at least one adverse event during surgery and a third of them are preventable. The occurrence of adverse events forces surgeons to implement corrective strategies and, thus, deviate from the standard surgical process. Therefore, it is clear that the automatic identification of adverse events is a major challenge for patient safety. In this paper, we have proposed a method enabling us to identify such deviations. We have focused on identifying surgeons' deviations from standard surgical processes due to surgical events rather than anatomic speci…
ICube, University of Strasbourg, CNRS（斯特拉斯堡大学ICube实验室，法国国家科学研究中心）；Fondazione Policlinico Universitario A. Gemelli IRCCS（A. Gemelli大学综合医院基金会IRCCS）；Department of Digestive and Endocrine Surgery, Nouvel Hôpital Civil, Hôpitaux Universitaires de Strasbourg（斯特拉斯堡大学医院新公民医院消化与内分泌外科）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡影像引导外科研究所（IHU-Strasbourg））；Institute for Research against Digestive Cancer (IRCAD)（消化系统癌症研究所（IRCAD））
Artificial intelligence is set to be deployed in operating rooms to improve surgical care. This early-stage clinical evaluation shows the feasibility of concurrently attaining real-time, high-quality predictions from several deep neural networks for endoscopic video analysis deployed for assistance during three laparoscopic cholecystectomies.
Attention-Aware Laparoscopic Image Desmoking Network with Lightness Embedding and Hybrid Guided Embedding
School of Life Science and Technology, Harbin Institute of Technology（哈尔滨工业大学生命科学与技术学院）；State Key Laboratory of Robotics and System, Harbin Institute of Technology（哈尔滨工业大学机器人技术与系统国家重点实验室）；State Key Laboratory of Robotics, Shenyang Institute of Automation（沈阳自动化研究所机器人学国家重点实验室）
The dominant visual servoing approaches in Minimally Invasive Surgery (MIS) follow single points or adapt the endoscope's field of view based on the surgical tools' distance. These methods rely on point positions with respect to the camera frame to infer a control policy. Deviating from the dominant methods, we formulate a robotic controller that allows for image-based visual servoing that requires neither explicit tool and camera positions nor any explicit image depth information. The proposed method relies on homography-based image registration, which changes the automation paradigm from poi…
Laparoscopic videos can be affected by different distortions which may impact the performance of surgery and introduce surgical errors. In this work, we propose a framework for automatically detecting and identifying such distortions and their severity using video quality assessment. There are three major contributions presented in this work (i) a proposal for a novel video enhancement framework for laparoscopic surgery; (ii) a publicly available database for quality assessment of laparoscopic videos evaluated by expert as well as non-expert observers and (iii) objective video quality assessme…
Université Sorbonne Paris Nord（索邦巴黎北大学）；Institut Galilée（伽利略研究所）；NTNU（挪威科技大学）
Video quality assessment is a challenging problem having a critical significance in the context of medical imaging. For instance, in laparoscopic surgery, the acquired video data suffers from different kinds of distortion that not only hinder surgery performance but also affect the execution of subsequent tasks in surgical navigation and robotic surgeries. For this reason, we propose in this paper neural network-based approaches for distortion classification as well as quality prediction. More precisely, a Residual Network (ResNet) based approach is firstly developed for simultaneous ranking a…
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；Fondazione Policlinico Universitario A. Gemelli IRCCS（杰梅利大学医院基金会）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡大学医院-图像引导手术研究所）；Institute for Research against Digestive Cancer (IRCAD)（消化癌症研究所）
Minimally invasive image-guided surgery heavily relies on vision. Deep learning models for surgical video analysis could therefore support visual tasks such as assessing the critical view of safety (CVS) in laparoscopic cholecystectomy (LC), potentially contributing to surgical safety and efficiency. However, the performance, reliability and reproducibility of such models are deeply dependent on the quality of data and annotations used in their development. Here, we present a protocol, checklists, and visual examples to promote consistent annotation of hepatocystic anatomy and CVS criteria. We…
Live Laparoscopic Video Retrieval with Compressed Uncertainty
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅里大学综合诊所基金会IRCCS）；IRCAD（法国消化外科研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）；Soochow University（苏州大学）
Objective: The computation of anatomical information and laparoscope position is a fundamental block of surgical navigation in Minimally Invasive Surgery (MIS). Recovering a dense 3D structure of surgical scene using visual cues remains a challenge, and the online laparoscopic tracking primarily relies on external sensors, which increases system complexity. Methods: Here, we propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is obtained. To reconstruct the 3D structure of the whole surgical environmen…
Heidelberg University Hospital（海德堡大学医院）；National Center for Tumor Diseases (NCT) Heidelberg（海德堡国家肿瘤疾病中心）；KARL STORZ SE & Co. KG（卡尔史托斯股份有限公司）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
PURPOSE: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data single-center dataset. In this work we investigated the generalizability of phase recognition algorithms in a multi-center setting including more difficult recognit…
