Computer-assisted surgery has been developed to enhance surgery correctness and safety. However, researchers and engineers suffer from limited annotated data to develop and train better algorithms. Consequently, the development of fundamental algorithms such as Simultaneous Localization and Mapping (SLAM) is limited. This article elaborates on the efforts of preparing the dataset for semantic segmentation, which is the foundation of many computer-assisted surgery mechanisms. Based on the Cholec80 dataset [3], we extracted 8,080 laparoscopic cholecystectomy image frames from 17 video clips in Cholec80 and annotated the images. The dataset is named CholecSeg8K and its total size is 3GB. Each of these images is annotated at pixel-level for thirteen classes, which are commonly founded in laparoscopic cholecystectomy surgery. CholecSeg8k is released under the license CC BY- NC-SA 4.0.
Automatic recognition of surgical phases in surgical videos is a fundamental task in surgical workflow analysis. In this report, we propose a Transformer-based method that utilizes calibrated confidence scores for a 2-stage inference pipeline, which dynamically switches between a baseline model and a separately trained transition model depending on the calibrated confidence level. Our method outperforms the baseline model on the Cholec80 dataset, and can be applied to a variety of action segmentation methods.
CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic Surgery
Kings College London, Surgical & Interventional Engineering（伦敦国王学院，外科与介入工程学院）；Kings College Hospital Denmark Hill, department（伦敦国王学院丹尼希尔医院，部门）；Department of Informatics, King’s College London（国王学院信息学院）；Tongji University, College of Electronic and Information Engineering（同济大学，电子与信息工程学院）
A Bayesian Approach to Weakly-supervised Laparoscopic Image Segmentation
Graduate School of Informatics, Nagoya University（名古屋大学情报学研究科）；Information Technology Center, Nagoya University（名古屋大学信息技术中心）；School of Information Science, Aichi Institute of Technology（爱知工业大学信息科学部）；Research Center of Medical Bigdata, National Institute of Informatics（国立信息学研究所医疗大数据研究中心）
ICube, University of Strasbourg, CNRS（斯特拉斯堡大学ICube实验室，法国国家科学研究中心）；Fondazione Policlinico Universitario A. Gemelli IRCCS（A. Gemelli大学综合医院基金会IRCCS）；Department of Digestive and Endocrine Surgery, Nouvel Hôpital Civil, Hôpitaux Universitaires de Strasbourg（斯特拉斯堡大学医院新公民医院消化与内分泌外科）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡影像引导外科研究所（IHU-Strasbourg））；Institute for Research against Digestive Cancer (IRCAD)（消化系统癌症研究所（IRCAD））
Artificial intelligence is set to be deployed in operating rooms to improve surgical care. This early-stage clinical evaluation shows the feasibility of concurrently attaining real-time, high-quality predictions from several deep neural networks for endoscopic video analysis deployed for assistance during three laparoscopic cholecystectomies.
Learning from Single Timestamps: Complexity Estimation in Laparoscopic Cholecystectomy
UCL Hawkes Institute, University College London（UCL哈维斯研究所，伦敦大学学院）；Medtronic plc.（美敦力公司）；Dept of Medical Physics & Biomedical Engineering, University College London（医学物理与生物医学工程系，伦敦大学学院）；Dept of Computer Science, University College London（计算机科学系，伦敦大学学院）
University College London（伦敦大学学院）；InstaDeep（InstaDeep公司）
Improving a semi-supervised image segmentation task has the option of adding more unlabelled images, labelling the unlabelled images or combining both, as neither image acquisition nor expert labelling can be considered trivial in most clinical applications. With a laparoscopic liver image segmentation application, we investigate the performance impact by altering the quantities of labelled and unlabelled training data, using a semi-supervised segmentation algorithm based on the mean teacher learning paradigm. We first report a significantly higher segmentation accuracy, compared with supervis…
Early Operative Difficulty Assessment in Laparoscopic Cholecystectomy via Snapshot-Centric Video Analysis
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（法国斯特拉斯堡大学，CNRS，INSERM，ICube，UMR7357）；IHU Strasbourg, Strasbourg, France（斯特拉斯堡IHU）；Università degli Studi di Milano General Surgery Department（米兰大学外科部）；University of Torino（都灵大学）；Department of Medical Surgical Science and Translational Medicine, Sant’Andrea Hospital, Sapienza University of Rome（罗马Sapienza大学医学与外科科学及转化医学系，圣安德烈医院）；Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy（罗马A.Gemelli大学医院研究院）；Università Cattolica del Sacro Cuore, Rome, Italy（罗马圣心大学）；University Hospital of Strasbourg, France（斯特拉斯堡大学医院）
hutom；Yonsei University College of Medicine（延世大学医学院）
Automated surgical instrument localization is an important technology to understand the surgical process and in order to analyze them to provide meaningful guidance during surgery or surgical index after surgery to the surgeon. We introduce a new dataset that reflects the kinematic characteristics of surgical instruments for automated surgical instrument localization of surgical videos. The hSDB(hutom Surgery DataBase)-instrument dataset consists of instrument localization information from 24 cases of laparoscopic cholecystecomy and 24 cases of robotic gastrectomy. Localization information for…
National University of Sciences and Technology（国家科技大学）；National Center for Artificial Intelligence（国家人工智能中心）
Autonomous surgical procedures, in particular minimal invasive surgeries, are the next frontier for Artificial Intelligence research. However, the existing challenges include precise identification of the human anatomy and the surgical settings, and modeling the environment for training of an autonomous agent. To address the identification of human anatomy and the surgical settings, we propose a deep learning based semantic segmentation algorithm to identify and label the tissues and organs in the endoscopic video feed of the human torso region. We present an annotated dataset, m2caiSeg, creat…
The Endoscapes Dataset for Surgical Scene Segmentation, Object Detection, and Critical View of Safety Assessment: Official Splits and Benchmark
ICube, University of Strasbourg, CNRS（ICube，斯特拉斯堡大学，CNRS）；IHU Strasbourg（斯特拉斯堡IHU）；Institute for Research against Digestive Cancer (IRCAD)（消化道癌症研究所）；Fondazione Policlinico Universitario A. Gemelli IRCCS（A.Gemelli大学附属医院）
Latent Graph Representations for Critical View of Safety Assessment
ICube（ICube实验室）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU-Strasbourg（斯特拉斯堡大学医院研究所）；Institute of Image-Guided Surgery（图像引导外科研究所）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅里大学综合诊所基金会IRCCS）；Institute for Research against Digestive Cancer (IRCAD)（消化道癌症研究所（IRCAD））
MIT（麻省理工学院）；Massachusetts General Hospital（麻省总医院）；Olympus Corporation（奥林巴斯公司）
Comprehension of surgical workflow is the foundation upon which artificial intelligence (AI) and machine learning (ML) holds the potential to assist intraoperative decision-making and risk mitigation. In this work, we move beyond mere identification of past surgical phases, into the prediction of future surgical steps and specification of the transitions between them. We use a novel Generative Adversarial Network (GAN) formulation to sample future surgical phases trajectories conditioned on past video frames from laparoscopic cholecystectomy (LC) videos and compare it to state-of-the-art appro…
Prior work demonstrated the ability of machine learning to automatically recognize surgical workflow steps from videos. However, these studies focused on only a single type of procedure. In this work, we analyze, for the first time, surgical step recognition on four different laparoscopic surgeries: Cholecystectomy, Right Hemicolectomy, Sleeve Gastrectomy, and Appendectomy. Inspired by the traditional apprenticeship model, in which surgical training is based on the Halstedian method, we paraphrase the "see one, do one, teach one" approach for the surgical intelligence domain as "train one, cla…
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；Fondazione Policlinico Universitario A. Gemelli IRCCS（杰梅利大学医院基金会）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡大学医院-图像引导手术研究所）；Institute for Research against Digestive Cancer (IRCAD)（消化癌症研究所）
Minimally invasive image-guided surgery heavily relies on vision. Deep learning models for surgical video analysis could therefore support visual tasks such as assessing the critical view of safety (CVS) in laparoscopic cholecystectomy (LC), potentially contributing to surgical safety and efficiency. However, the performance, reliability and reproducibility of such models are deeply dependent on the quality of data and annotations used in their development. Here, we present a protocol, checklists, and visual examples to promote consistent annotation of hepatocystic anatomy and CVS criteria. We…
National and Kapodistrian University of Athens（雅典国家与卡波迪斯特里安大学）
An important task at the onset of a laparoscopic cholecystectomy (LC) operation is the inspection of gallbladder (GB) to evaluate the thickness of its wall, presence of inflammation and extent of fat. Difficulty in visualization of the GB wall vessels may be due to the previous factors, potentially as a result of chronic inflammation or other diseases. In this paper we propose a multiple-instance learning (MIL) technique for assessment of the GB wall vascularity via computer-vision analysis of images from LC operations. The bags correspond to a labeled (low vs. high) vascularity dataset of 181…
ICube（ICube实验室）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；IRCAD（法国消化外科研究所）
Accurate surgery duration estimation is necessary for optimal OR planning, which plays an important role in patient comfort and safety as well as resource optimization. It is, however, challenging to preoperatively predict surgery duration since it varies significantly depending on the patient condition, surgeon skills, and intraoperative situation. In this paper, we propose a deep learning pipeline, referred to as RSDNet, which automatically estimates the remaining surgery duration (RSD) intraoperatively by using only visual information from laparoscopic videos. Previous state-of-the-art appr…
ICube, University of Strasbourg, CNRS（斯特拉斯堡大学 ICube 实验室、法国国家科学研究中心）；Fondazione Policlinico Universitario A. Gemelli IRCCS（A. Gemelli 大学综合医院基金会 IRCCS）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡 IHU 影像引导外科研究所）；Institute for Research against Digestive Cancer (IRCAD)（消化道癌症研究院（IRCAD））
A major obstacle to building models for effective semantic segmentation, and particularly video semantic segmentation, is a lack of large and well annotated datasets. This bottleneck is particularly prohibitive in highly specialized and regulated fields such as medicine and surgery, where video semantic segmentation could have important applications but data and expert annotations are scarce. In these settings, temporal clues and anatomical constraints could be leveraged during training to improve performance. Here, we present Temporally Constrained Neural Networks (TCNN), a semi-supervised fr…
Tecnológico de Monterrey（蒙特雷科技大学）；University of Oxford（牛津大学）
Image-based tracking of laparoscopic instruments plays a fundamental role in computer and robotic-assisted surgeries by aiding surgeons and increasing patient safety. Computer vision contests, such as the Robust Medical Instrument Segmentation (ROBUST-MIS) Challenge, seek to encourage the development of robust models for such purposes, providing large, diverse, and annotated datasets. To date, most of the existing models for instance segmentation of medical instruments were based on two-stage detectors, which provide robust results but are nowhere near to the real-time (5 frames-per-second (fp…
German Cancer Research Center（德国癌症研究中心）；University of Heidelberg（海德堡大学）；Heidelberg University Hospital（海德堡大学医院）；caresyntax；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；SimulaMet；Arctic University of Norway（挪威北极大学）；Oslo Metropolitan University（奥斯都会大学）；University of Electronic Science and Technology of China（电子科技大学）；National Center for Tumor Diseases（国家肿瘤疾病中心）；Technische Universität Dresden（德累斯顿工业大学）；Helmholtz-Zentrum Dresden - Rossendorf（德累斯顿-罗森多夫亥姆霍兹中心）；Universidad de los Andes（安第斯大学）；Peking University（北京大学）；Xiamen University（厦门大学）
Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should ge…
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University（海德堡大学）；University of Calabria（卡拉布里亚大学）；Scuola Superiore Sant’Anna（圣安娜高等科学院）；Heidelberg University Hospital（海德堡大学医院）
Challenges have become the state-of-the-art approach to benchmark image analysis algorithms in a comparative manner. While the validation on identical data sets was a great step forward, results analysis is often restricted to pure ranking tables, leaving relevant questions unanswered. Specifically, little effort has been put into the systematic investigation on what characterizes images in which state-of-the-art algorithms fail. To address this gap in the literature, we (1) present a statistical framework for learning from challenges and (2) instantiate it for the specific task of instrument …
Computer Science and Artificial Intelligence Laboratory（计算机科学与人工智能实验室）；Massachusetts General Hospital（麻省总医院）；Toyota Research Institute（丰田研究院）；Olympus Corporation（奥林巴斯公司）
Analyzing surgical workflow is crucial for surgical assistance robots to understand surgeries. With the understanding of the complete surgical workflow, the robots are able to assist the surgeons in intra-operative events, such as by giving a warning when the surgeon is entering specific keys or high-risk phases. Deep learning techniques have recently been widely applied to recognizing surgical workflows. Many of the existing temporal neural network models are limited in their capability to handle long-term dependencies in the data, instead, relying upon the strong performance of the underlyin…
Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR)
Regensburg Medical Image Computing (ReMIC), OTH Regensburg（雷根斯堡医学影像计算中心（ReMIC））；Regensburg Center of Biomedical Engineering (RCBE), OTH Regensburg（雷根斯堡生物医学工程中心（RCBE））；Regensburg University（雷根斯堡大学）；Regensburg Center of Health Sciences（雷根斯堡健康科学中心）；Technical University of Munich (TUM)（慕尼黑技术大学）；TUM University Hospital（慕尼黑技术大学医院）；Biomedical Image Analysis Group, Department of Computing, Imperial College London（生物医学影像分析组，计算部门，伦敦帝国学院）；Research Group MITI, TUM University Hospital, School of Medicine（MITI研究组，慕尼黑技术大学医院，医学学院）；Department of Surgery, TUM University Hospital, School of Medicine（外科部门，慕尼黑技术大学医院，医学学院）
LapFM: A Laparoscopic Segmentation Foundation Model via Hierarchical Concept Evolving Pre-training
School of Computer Science, University of Lincoln, UK（英国林肯大学计算机科学学院）；University of Nottingham, UK（英国诺丁汉大学）；University of Nottingham Ningbo China, China（中国宁波诺丁汉大学）；University of Strasbourg, France（法国斯特拉斯堡大学）；Technical University of Munich, Germany（德国慕尼黑技术大学）；Graduate School of Information, Production and Systems, Waseda University, Japan（日本早稻田大学信息、生产与系统研究生院）；Department of Gastrointestinal Surgery, The Second Qilu Hospital, Shandong University, China（中国山东大学第二齐鲁医院胃肠外科）；School of Engineering and Physical Science, University of Lincoln, Lincoln LN6 7TS, UK（英国林肯大学工程与物理科学学院）；Yale University, New Haven, CT 06510, USA（美国耶鲁大学）
Adaptive transfer learning for surgical tool presence detection in laparoscopic videos through gradual freezing fine-tuning
Institutes of Innovation for Future Society（未来社会创新研究所）；Nagoya University（名古屋大学）；Department of Micro-Nano Mechanical Science and Engineering（微纳米机械科学与工程系）
SurgiATM: A Physics-Guided Plug-and-Play Model for Deep Learning-Based Smoke Removal in Laparoscopic Surgery
School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Institute of Medical Robotics, School of Biomedical Engineering, Shanghai Jiao Tong University（上海交通大学生物医学工程学院医学机器人研究所）；Department of Radiology, Brigham and Women's Hospital, and Harvard Medical School（哈佛医学院放射科、布里洛女性医院）
One-shot skill assessment in high-stakes domains with limited data via meta learning
Florida A&M University and The Florida State University（佛罗里达农工大学与佛罗里达州立大学）；University at Buffalo（布法罗大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；U.S. Army Combat Capabilities Development Command Soldier Center STTC（美国陆军作战能力发展司令部士兵中心 STTC）
Exploring Optical Flow Inclusion into nnU-Net Framework for Surgical Instrument Segmentation
Life and Health Sciences Research Institute (ICVS)（生命与健康科学研究所（ICVS））；School of Medicine, University of Minho（米尼奥大学医学院）；ICVS/3B’s - PT Government Associate Laboratory（ICVS/3B's - 葡萄牙政府附属实验室）；School of Technology, IPCA（IPCA 技术学院）；LASI – Associate Laboratory of Intelligent Systems（LASI——智能系统附属实验室）；Algoritmi Center, School of Engineering, University of Minho（米尼奥大学工程学院 Algoritmi 中心）；KARL STORZ SE & Co. KG（卡尔史托斯有限两合公司）
ICube, University of Strasbourg, CNRS, IHU Strasbourg（斯特拉斯堡大学ICube研究所、法国国家科学研究中心、斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg, IRCAD, IHU Strasbourg（斯特拉斯堡大学医院、斯特拉斯堡 IRCAD 研究所、斯特拉斯堡大学医院研究所）
Real-time algorithms for automatically recognizing surgical phases are needed to develop systems that can provide assistance to surgeons, enable better management of operating room (OR) resources and consequently improve safety within the OR. State-of-the-art surgical phase recognition algorithms using laparoscopic videos are based on fully supervised training. This limits their potential for widespread application, since creation of manual annotations is an expensive process considering the numerous types of existing surgeries and the vast amount of laparoscopic videos available. In this work…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Heidelberg（海德堡大学）
Computer-assisted surgery (CAS) aims to provide the surgeon with the right type of assistance at the right moment. Such assistance systems are especially relevant in laparoscopic surgery, where CAS can alleviate some of the drawbacks that surgeons incur. For many assistance functions, e.g. displaying the location of a tumor at the appropriate time or suggesting what instruments to prepare next, analyzing the surgical workflow is a prerequisite. Since laparoscopic interventions are performed via endoscope, the video signal is an obvious sensor modality to rely on for workflow analysis. Image-ba…
Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge
Regensburg Medical Image Computing (ReMIC), OTH Regensburg（雷根萨大学医学影像计算中心）；Research Group MITI, TUM University Hospital, School of Medicine and Health（技术大学慕尼黑大学医院MITI研究组）；Regensburg Center of Biomedical Engineering (RCBE), OTH Regensburg（雷根萨生物医学工程研究中心）；Regensburg University（雷根萨大学）；Regensburg Center of Health Sciences and Technology (RCHST), OTH Regensburg（雷根萨健康科学与技术研究中心）；AI Centre of Excellence, Medtronic Ltd.（医学影像人工智能卓越中心）；Engineering Sciences, University College London（伦敦大学学院工程科学系）；Augmented Intelligence Lab, Kyung Hee University（庆熙大学增强智能实验室）；University of Minho, Braga（明霍大学）；Jmees Inc.（Jmees公司）；School of Sciences, São Paulo State University (UNESP), Bauru（圣保罗州立大学科学学院）；KIST HARILAB, Center for Humanoid Research, Artificial Intelligence and Robot Institute, Korea Institute of Science and Technology (KIST)（韩国科学技术院HARILAB中心）；King's College London（伦敦国王学院）；The Chinese University of Hong Kong（香港中文大学）；Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ)（德国癌症研究中心智能医疗系统部门）；Muroran Institute of Technology, Hokkaido（北海道Muroran技术学院）；Niigata University of Health and Welfare（Niigata医疗福利大学）；Konica Minolta, Inc.（东宝株式会社）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology（香港科技大学化学与生物工程系）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, Shenzhen（香港科技大学深圳-香港协同创新研究院）；Center for Research and Formation in Artificial Intelligence (CinfonIA), Los Andes University, Bogota（安第斯大学人工智能研究与培养中心）；Department of Biomedical Engineering, Medical Image Analysis Group, Eindhoven University of Technology（埃因霍温理工大学生物医学工程系）；Institute of Information Technology (ITEC), Klagenfurt University（克雷格夫大学信息技术研究所）；Center for Tactile Internet with Human-in-the-loop (CeTI), TU Dresden（德累斯顿技术大学触觉互联网中心）
Surgical Anatomy Recognition with Context Learning using Foundation Representations
Department of Biomedical Engineering, Medical Image Analysis, Eindhoven University of Technology（埃因霍温理工大学，生物医学工程系，医学图像分析）；Department of Electrical Engineering, Architectures for Reliable Image Analysis, Eindhoven University of Technology（埃因霍温理工大学，电气工程系，可靠图像分析架构）；Department of Surgery, University Medical Center Utrecht（乌得勒支大学医学中心，外科）；Department of Oncological Urology, University Medical Center Utrecht（乌得勒支大学医学中心，肿瘤泌尿外科）；Department of Urology, Catharina Hospital（凯瑟琳娜医院，泌尿外科）；Department of Electrical Engineering, Mobile Perception Systems Lab, Eindhoven University of Technology（埃因霍温理工大学，电气工程系，移动感知系统实验室）
Preserving Privacy in Surgical Video Analysis Using Artificial Intelligence: A Deep Learning Classifier to Identify Out-of-Body Scenes in Endoscopic Videos
IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Bern（伯尔尼大学）；Inselspital, Bern University Hospital（伯尔尼大学医院因塞尔医院）；University of Strasbourg（斯特拉斯堡大学）；ICube（ICube实验室）；CNRS（法国国家科学研究中心）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（天主教大学 Gemelli 医院基金会）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Heidelberg（海德堡大学）
During laparoscopic surgery, context-aware assistance systems aim to alleviate some of the difficulties the surgeon faces. To ensure that the right information is provided at the right time, the current phase of the intervention has to be known. Real-time locating and classification the surgical tools currently in use are key components of both an activity-based phase recognition and assistance generation. In this paper, we present an image-based approach that detects and classifies tools during laparoscopic interventions in real-time. First, potential instrument bounding boxes are detected us…
ICube（ICube研究所）；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；IRCAD（消化癌症研究与治疗研究所）
Surgical tool localization is an essential task for the automatic analysis of endoscopic videos. In the literature, existing methods for tool localization, tracking and segmentation require training data that is fully annotated, thereby limiting the size of the datasets that can be used and the generalization of the approaches. In this work, we propose to circumvent the lack of annotated data with weak supervision. We propose a deep architecture, trained solely on image level annotations, that can be used for both tool presence detection and localization in surgical videos. Our architecture re…
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London（伦敦大学学院威康/EPSRC介入与外科科学中心）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
Semantic tool segmentation in surgical videos is important for surgical scene understanding and computer-assisted interventions as well as for the development of robotic automation. The problem is challenging because different illumination conditions, bleeding, smoke and occlusions can reduce algorithm robustness. At present labelled data for training deep learning models is still lacking for semantic surgical instrument segmentation and in this paper we show that it may be possible to use robot kinematic data coupled with laparoscopic images to alleviate the labelling problem. We propose a ne…
University Hospital Heidelberg（海德堡大学医院）；Otto-von-Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）
Minimally Invasive Surgery (MIS) techniques have gained rapid popularity among surgeons since they offer significant clinical benefits including reduced recovery time and diminished post-operative adverse effects. However, conventional endoscopic systems output monocular video which compromises depth perception, spatial orientation and field of view. Suturing is one of the most complex tasks performed under these circumstances. Key components of this tasks are the interplay between needle holder and the surgical needle. Reliable 3D localization of needle and instruments in real time could be u…
The Chinese University of Hong Kong（香港中文大学）；Soochow University（苏州大学）；Prince of Wales Hospital（威尔斯亲王医院）
Computer-assisted minimally invasive surgery has great potential in benefiting modern operating theatres. The video data streamed from the endoscope provides rich information to support context-awareness for next-generation intelligent surgical systems. To achieve accurate perception and automatic manipulation during the procedure, learning based technique is a promising way, which enables advanced image analysis and scene understanding in recent years. However, learning such models highly relies on large-scale, high-quality, and multi-task labelled data. This is currently a bottleneck for the…
National Center for Tumor Diseases（国家肿瘤疾病中心）；University College London（伦敦大学学院）；Mannheim University of Applied Sciences（曼海姆应用科学大学）；German Cancer Research Center（德国癌症研究中心）
In the medical domain, the lack of large training data sets and benchmarks is often a limiting factor for training deep neural networks. In contrast to expensive manual labeling, computer simulations can generate large and fully labeled data sets with a minimum of manual effort. However, models that are trained on simulated data usually do not translate well to real scenarios. To bridge the domain gap between simulated and real laparoscopic images, we exploit recent advances in unpaired image-to-image translation. We extent an image-to-image translation method to generate a diverse multitude o…
National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心（NCT））；Intuitive Surgical（直觉外科公司）；Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；CHU Grenoble Alpes（格勒诺布尔阿尔卑斯大学医院）；Grenoble INP（格勒诺布尔综合理工学院）；TIMC-IMAG（TIMC-IMAG实验室）；Centre for Medical Image Computing（医学影像计算中心）；University College London（伦敦大学学院）；Department of General, Visceral and Transplant Surgery, University of Heidelberg（海德堡大学普通、内脏和移植外科系）；ARTORG Center, University of Bern（伯尔尼大学ARTORG中心）；Computer Aided Medical Procedures, Johns Hopkins University（约翰霍普金斯大学计算机辅助医疗程序）；Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院（KIT）人机工程与机器人研究所）
Intraoperative segmentation and tracking of minimally invasive instruments is a prerequisite for computer- and robotic-assisted surgery. Since additional hardware like tracking systems or the robot encoders are cumbersome and lack accuracy, surgical vision is evolving as promising techniques to segment and track the instruments using only the endoscopic images. However, what is missing so far are common image data sets for consistent evaluation and benchmarking of algorithms against each other. The paper presents a comparative validation study of different vision-based methods for instrument s…
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University Hospital（海德堡大学医院）；University of Heidelberg（海德堡大学）；National Center for Tumor Diseases, Partner Site Dresden（国家肿瘤疾病中心德累斯顿分中心）；HIDSS4Health – Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；University of Calabria（卡拉布里亚大学）；understandAI GmbH（understandAI 有限公司）；University of Tübingen（蒂宾根大学）；BMW Group（宝马集团）；KARL STORZ SE & Co. KG（卡尔史托斯公司）；TU Dresden（德累斯顿工业大学）
Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a speci…
Generative data augmentation for biliary tract detection on intraoperative images
C.R.E.A.T.E. Consorzio di Ricerca per l’Energia, l’Automazione e le Tecnologie dell’Elettromagnetismo, Università degli Studi di Napoli Federico II（C.R.E.A.T.E.研究会，那不勒斯费德里科二世大学）；Department of Information Technology and Electrical Engineering, Università degli Studi di Napoli Federico II（信息科技与电气工程系，那不勒斯费德里科二世大学）；Department of Information Engineering, University of Padova（信息工程系，帕多瓦大学）；General Surgical Unit, Department of Public Health, Università degli Studi di Napoli Federico II（外科部，公共卫生系，那不勒斯费德里科二世大学）
