Accurate and real-time surgical instrument segmentation is important in the endoscopic vision of robot-assisted surgery, and significant challenges are posed by frequent instrument-tissue contacts and continuous change of observation perspective. For these challenging tasks more and more deep neural networks (DNN) models are designed in recent years. We are motivated to propose a general embeddable approach to improve these current DNN segmentation models without increasing the model parameter number. Firstly, observing the limited rotation-invariance performance of DNN, we proposed the Multi-Angle Feature Aggregation (MAFA) method, leveraging active image rotation to gain richer visual cues and make the prediction more robust to instrument orientation changes. Secondly, in the end-to-end training stage, the auxiliary contour supervision is utilized to guide the model to learn the boundary awareness, so that the contour shape of segmentation mask is more precise. The proposed method is validated with ablation experiments on the novel Sinus-Surgery datasets collected from surgeons' operations, and is compared to the existing methods on a public dataset collected with a da Vinci Xi Robot.
Asan Medical Center（峨山医疗中心）；Johns Hopkins University（约翰斯·霍普金斯大学）；Chungnam National University School of Medicine（忠南大学医学院）
Some endoscopic surgeries require a surgeon to hold the endoscope with one hand and the surgical instruments with the other hand to perform the actual surgery with correct vision. Recent technical advances in deep learning as well as in robotics can introduce robotics to these endoscopic surgeries. This can have numerous advantages by freeing one hand of the surgeon, which will allow the surgeon to use both hands and to use more intricate and sophisticated techniques. Recently, deep learning with convolutional neural network achieves state-of-the-art results in computer vision. Therefore, the …
Benchmarking Endoscopic Surgical Image Restoration and Beyond
The Chinese University of Hong Kong（香港中文大学）；Dalian University of Technology（大连理工大学）；Southern Medical University（南方医科大学）；Xidian University（西安电子科技大学）；Wuhan University（武汉大学）
TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation
Birmingham City University（伯明翰城市大学）；University Hospitals Birmingham（伯明翰大学医院）；University of Birmingham（伯明翰大学）；University of the West of England（西英格兰大学）；University of Hertfordshire（赫特福德大学）；University of Bristol（布里斯托大学）；Qatar University（卡塔尔大学）
Accurate and robust tracking and reconstruction of the surgical scene is a critical enabling technology toward autonomous robotic surgery. Existing algorithms for 3D perception in surgery mainly rely on geometric information, while we propose to also leverage semantic information inferred from the endoscopic video using image segmentation algorithms. In this paper, we present a novel, comprehensive surgical perception framework, Semantic-SuPer, that integrates geometric and semantic information to facilitate data association, 3D reconstruction, and tracking of endoscopic scenes, benefiting dow…
EndoLive: Real-Time Style Transfer for Endoscopic Endonasal Skull Base Surgical Video
University of Pittsburgh（匹兹堡大学）；University of Pittsburgh School of Computing and Information（匹兹堡大学计算与信息学院）；School of Computing and Information（计算与信息学院）
ICube, University of Strasbourg, CNRS（斯特拉斯堡大学ICube研究所，法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅利大学综合医院基金会IRCCS）；IRCAD France（法国消化癌症研究研究所）
Out of all existing frameworks for surgical workflow analysis in endoscopic videos, action triplet recognition stands out as the only one aiming to provide truly fine-grained and comprehensive information on surgical activities. This information, presented as <instrument, verb, target> combinations, is highly challenging to be accurately identified. Triplet components can be difficult to recognize individually; in this task, it requires not only performing recognition simultaneously for all three triplet components, but also correctly establishing the data association between them. To achieve …
University of Washington（华盛顿大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Deep learning-based methods have achieved promising results on surgical instrument segmentation. However, the high computation cost may limit the application of deep models to time-sensitive tasks such as online surgical video analysis for robotic-assisted surgery. Moreover, current methods may still suffer from challenging conditions in surgical images such as various lighting conditions and the presence of blood. We propose a novel Multi-frame Feature Aggregation (MFFA) module to aggregate video frame features temporally and spatially in a recurrent mode. By distributing the computation load…
SAGS: Self-Adaptive Alias-Free Gaussian Splatting for Dynamic Surgical Endoscopic Reconstruction
University of Technology Sydney, Australia（悉尼技术大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China（深圳先进技术研究所）；Shenyang Institute of Automation, Chinese Academy of Sciences, China（沈阳自动化研究所）
CPKD: Clinical Prior Knowledge-Constrained Diffusion Models for Surgical Phase Recognition in Endoscopic Submucosal Dissection
School of Biomedical Engineering（生物医学工程学院）；Division of Gastroenterology and Hepatology, Shanghai Institute of Digestive Disease, NHC Key Laboratory of Digestive Diseases, Renji Hospital（消化内科与肝病科、上海消化疾病研究所、国家消化疾病临床医学研究中心、仁济医院）；College of Media Engineering（媒体工程学院）；Aier Institute of Digital Ophthalmology and Visual Science（数字眼科与视觉科学研究院）
Preserving Privacy in Surgical Video Analysis Using Artificial Intelligence: A Deep Learning Classifier to Identify Out-of-Body Scenes in Endoscopic Videos
IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Bern（伯尔尼大学）；Inselspital, Bern University Hospital（伯尔尼大学医院因塞尔医院）；University of Strasbourg（斯特拉斯堡大学）；ICube（ICube实验室）；CNRS（法国国家科学研究中心）；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（天主教大学 Gemelli 医院基金会）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
SPRMamba: Surgical Phase Recognition for Endoscopic Submucosal Dissection with Mamba
School of Biomedical Engineering（生物医学工程学院）；Division of Gastroenterology and Hepatology, Shanghai Institute of Digestive Disease, NHC Key Laboratory of Digestive Diseases, Renji Hospital（消化内科与肝病科、上海消化疾病研究所、国家消化疾病临床医学研究中心、仁济医院）；Aier Institute of Digital Ophthalmology and Visual Science（爱尔数字眼科与视觉科学研究院）；College of Media Engineering（媒体工程学院）
EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Theory Lab, Central Research Institute, 2012 Labs, Huawei Technologies Co. Ltd.（华为技术有限公司中央研究院理论实验室2012实验室）；Chair of Computer Aided Procedures (CAMP), Technical University of Munich（慕尼黑工业大学计算机辅助程序教席）；Department of Gastroenterology, Qilu Hospital of Shandong University（山东大学齐鲁医院消化内科）
CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Qilu Hospital of Shandong University（山东大学齐鲁医院）；National University o（国立大学）
Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心）；National University of Singapore（新加坡国立大学）
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information…
Zuse Institute Berlin (ZIB)（柏林祖斯研究所）；TU Darmstadt（达姆施塔特工业大学）
Purpose: Segmentation of surgical instruments in endoscopic videos is essential for automated surgical scene understanding and process modeling. However, relying on fully supervised deep learning for this task is challenging because manual annotation occupies valuable time of the clinical experts. Methods: We introduce a teacher-student learning approach that learns jointly from annotated simulation data and unlabeled real data to tackle the erroneous learning problem of the current consistency-based unsupervised domain adaptation framework. Results: Empirical results on three datasets highlig…
Geo-RepNet: Geometry-Aware Representation Learning for Surgical Phase Recognition in Endoscopic Submucosal Dissection
Department of Electronic Engineering, The Chinese University of Hong Kong（电子工程系，香港中文大学）；Theory Lab, Central Research Institute, 2012 Labs, Huawei Technologies Co. Ltd.（理论实验室，中央研究院，2012实验室，华为技术有限公司）
State-Change Learning for Prediction of Future Events in Endoscopic Videos
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学，法国国家科学研究中心，法国国家医学研究院，ICube，UMR7357，法国）；University Hospital of Strasbourg, France（斯特拉斯堡大学医院，法国）
Surgical tool segmentation in endoscopic images is an important problem: it is a crucial step towards full instrument pose estimation and it is used for integration of pre- and intra-operative images into the endoscopic view. While many recent approaches based on convolutional neural networks have shown great results, a key barrier to progress lies in the acquisition of a large number of manually-annotated images which is necessary for an algorithm to generalize and work well in diverse surgical scenarios. Unlike the surgical image data itself, annotations are difficult to acquire and may be o…
Indian Institute of Technology Delhi（印度理工学院德里分校）；All India Institute of Medical Sciences（全印度医学科学研究所）
Neuro-endoscopy is a challenging minimally invasive neurosurgery that requires surgical skills to be acquired using training methods different from the existing apprenticeship model. There are various training systems developed for imparting fundamental technical skills in laparoscopy where as limited systems for neuro-endoscopy. Neuro-Endo-Trainer was a box-trainer developed for endo-nasal transsphenoidal surgical skills training with video based offline evaluation system. The objective of the current study was to develop a modified version (Neuro-Endo-Trainer-Online Assessment System (NET-OA…
Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
The Chinese University of Hong Kong (CUHK)（香港中文大学）；University College London（伦敦大学学院）；Shun Hing Institute of Advanced Engineering, CUHK（香港中文大学信兴高等工程研究所）；Shenzhen Research Institute, CUHK（香港中文大学深圳研究院）
Registering Neural 4D Gaussians for Endoscopic Surgery
The Chinese University of Hong Kong (CUHK)（香港中文大学）；Shenzhen Research Institute, CUHK（香港中文大学深圳研究院）；KTH, Kungliga Tekniska högskolan Royal Institute of Technology（瑞典皇家理工学院）；Hong Kong Applied Science and Technology Research Institute Company Limited（香港应用科技研究院有限公司）
Challenging Vision-Language Models with Surgical Data: A New Dataset and Broad Benchmarking Study
organization= German Cancer Research Center (DKFZ) Heidelberg, Div. Intelligent Medical Systems , country= Germany；organization= National Center for Tumor Diseases (NCT), NCT Heidelberg , country= Germany；Data Science School for Health , country= Germany；organization= Medical Faculty, Heidelberg University , country= Germany；organization= Faculty of Mathematics；Computer Science, Heidelberg University , country= Germany；organization= Weldon School of Biomedical Engineering, Purdue University , city= West Lafayette , state= IN , country= USA；organization= Regenstrief Center for Healthcare Engineering (RCHE), Purdue University , city= West Lafayette , state= IN , country= USA；organization= Department of Biostatistics；Health Data Science, Richard M. Fairbanks School of Public Health, Indiana University School of Medicine , city= Indianapolis , state= IN , country= USA；organization= Department of Surgery, Indiana University School of Medicine , city= Indianapolis , state= IN , country= USA；organization= Department of Visceral, Thoracic；Vascular Surgery, University Hospital；Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology , city= Dresden , country= Germany
Northwick Park Hospital（诺斯威克公园医院）；German Cancer Research Center（德国癌症研究中心）；Imperial College London（帝国理工学院）；University College London（伦敦大学学院）
Intra-operative measurements of tissue shape and multi/ hyperspectral information have the potential to provide surgical guidance and decision making support. We report an optical probe based system to combine sparse hyperspectral measurements and spectrally-encoded structured lighting (SL) for surface measurements. The system provides informative signals for navigation with a surgical interface. By rapidly switching between SL and white light (WL) modes, SL information is combined with structure-from-motion (SfM) from white light images, based on SURF feature detection and Lucas-Kanade (LK) o…
Surgical robots are used to perform minimally invasive surgery and alleviate much of the burden imposed on surgeons. Our group has developed a surgical robot to aid in the removal of tumors at the base of the skull via access through the nostrils. To avoid injuring the patients, a collision-avoidance algorithm that depends on having an accurate model for the poses of the instruments' shafts is used. Given that the model's parameters can change over time owing to interactions between instruments and other disturbances, the online estimation of the poses of the instrument's shaft is essential. I…
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), UCL（伦敦大学学院威康/EPSRC介入与外科科学中心）；National Center for Tumor Diseases (NCT) Dresden（德累斯顿国家肿瘤疾病中心）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
In computer vision, reference datasets have been highly successful in promoting algorithmic development in stereo reconstruction. Surgical scenes gives rise to specific problems, including the lack of clear corner features, highly specular surfaces and the presence of blood and smoke. Publicly available datasets have been produced using CT and either phantom images or biological tissue samples covering a relatively small region of the endoscope field-of-view. We present a stereo-endoscopic reconstruction validation dataset based on CT (SERV-CT). Two {\it ex vivo} small porcine full torso cadav…
The Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins Medical Institutions（约翰斯·霍普金斯医学机构）；Laboratory for Computational Sensing and Robotics, The Johns Hopkins University（约翰斯·霍普金斯大学计算感知与机器人实验室）
In this work, we explore whether it is possible to learn representations of endoscopic video frames to perform tasks such as identifying surgical tool presence without supervision. We use a maximum mean discrepancy (MMD) variational autoencoder (VAE) to learn low-dimensional latent representations of endoscopic videos and manipulate these representations to distinguish frames containing tools from those without tools. We use three different methods to manipulate these latent representations in order to predict tool presence in each frame. Our fully unsupervised methods can identify whether end…
DGGAN: Degradation Guided Generative Adversarial Network for Real-time Endoscopic Video Enhancement
Department of Mechanical Engineering（机械工程系）；State Key Laboratory of Tribology in Advanced Equipment（先进设备摩擦学国家重点实验室）；Beijing Key Laboratory of Transformative High-end Manufacturing Equipment and Technology（北京 transformative 高端制造装备与技术重点实验室）；Tsinghua University（清华大学）
EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery
Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）；School of Advanced Manufacturing, Fuzhou University（福州大学先进制造学院）
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院）；Politecnico di Milano（米兰理工大学）
Automatic surgical instrument segmentation of endoscopic images is a crucial building block of many computer-assistance applications for minimally invasive surgery. So far, state-of-the-art approaches completely rely on the availability of a ground-truth supervision signal, obtained via manual annotation, thus expensive to collect at large scale. In this paper, we present FUN-SIS, a Fully-UNsupervised approach for binary Surgical Instrument Segmentation. FUN-SIS trains a per-frame segmentation model on completely unlabelled endoscopic videos, by solely relying on implicit motion information an…
Robust Surgical Tools Detection in Endoscopic Videos with Noisy Data
Information Technology University（信息技术大学）；University of Bristol（布里斯托尔大学）；Bristol Heart Institute（布里斯托尔心脏研究所）；University of the West of England（西英格兰大学）；Qatar University（卡塔尔大学）
Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo Images
Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology（医学技术与智能系统研究所，汉堡技术大学）；Department of Otorhinolaryngology, Head and Neck Surgery and Oncology, University Medical Center Hamburg-Eppendorf（耳鼻喉头颈外科与肿瘤学部，汉堡艾本多夫大学医学中心）；Martini-Klinik Prostate Cancer Center, University Medical Center Hamburg-Eppendorf（马尔廷诊所前列腺癌中心，汉堡艾本多夫大学医学中心）；Department of Urology, University Medical Center Hamburg-Eppendorf（泌尿外科，汉堡艾本多夫大学医学中心）
本文提出一种利用2D Tracking Any Point网络结合两个CoTracker模型实现内窥镜立体图像无标记3D组织追踪的新方法，实验验证了其在复杂手术场景中的准确性与潜力。
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）
We present a robust deep learning based 6 degrees-of-freedom (DoF) localization system for endoscopic capsule robots. Our system mainly focuses on localization of endoscopic capsule robots inside the GI tract using only visual information captured by a mono camera integrated to the robot. The proposed system is a 23-layer deep convolutional neural network (CNN) that is capable to estimate the pose of the robot in real time using a standard CPU. The dataset for the evaluation of the system was recorded inside a surgical human stomach model with realistic surface texture, softness, and surface l…
Robotic surgery has been proven to offer clear advantages during surgical procedures, however, one of the major limitations is obtaining haptic feedback. Since it is often challenging to devise a hardware solution with accurate force feedback, we propose the use of "visual cues" to infer forces from tissue deformation. Endoscopic video is a passive sensor that is freely available, in the sense that any minimally-invasive procedure already utilizes it. To this end, we employ deep learning to infer forces from video as an attractive low-cost and accurate alternative to typically complex and expe…
Endo3R: Unified Online Reconstruction from Dynamic Monocular Endoscopic Video
The Chinese University of Hong Kong（香港中文大学）；Hong Kong Center for Logistics Robotics（香港物流机器人中心）；Johns Hopkins University（约翰斯·霍普金斯大学）；Shanghai Jiao Tong University（上海交通大学）
The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Reconstructing the scene of robotic surgery from the stereo endoscopic video is an important and promising topic in surgical data science, which potentially supports many applications such as surgical visual perception, robotic surgery education and intra-operative context awareness. However, current methods are mostly restricted to reconstructing static anatomy assuming no tissue deformation, tool occlusion and de-occlusion, and camera movement. However, these assumptions are not always satisfied in minimal invasive robotic surgeries. In this work, we present an efficient reconstruction pipel…
Center for Research and Formation in Artificial Intelligence, Universidad de los Andes（安第斯大学人工智能研究与培训中心）
We study the task of semantic segmentation of surgical instruments in robotic-assisted surgery scenes. We propose the Instance-based Surgical Instrument Segmentation Network (ISINet), a method that addresses this task from an instance-based segmentation perspective. Our method includes a temporal consistency module that takes into account the previously overlooked and inherent temporal information of the problem. We validate our approach on the existing benchmark for the task, the Endoscopic Vision 2017 Robotic Instrument Segmentation Dataset, and on the 2018 version of the dataset, whose anno…
