LighTDiff: Surgical Endoscopic Image Low-Light Enhancement with T-Diffusion
The Chinese University of Hong Kong (CUHK)（香港中文大学）；Shenzhen Research Institute, CUHK（香港中文大学深圳研究院）；The University of Sydney（悉尼大学）；Qilu Hospital of Shandong University（山东大学齐鲁医院）
内窥镜在外科手术中的应用进展面临着光照不足等挑战。深度学习，尤其是去噪扩散概率模型（Denoising Diffusion Probabilistic Model, DDPM），在医学领域的低光照图像增强方面具有应用前景。然而，DDPM计算量大、速度慢，限制了其在实际医疗场景中的应用。为弥补这一不足，我们提出了一种轻量级DDPM，命名为LighTDiff。该模型采用T形架构，通过低分辨率图像捕获全局结构信息，并在后续去噪步骤中逐步恢复细节。我们进一步对模型进行剪枝，在保留性能的同时大幅缩减模型尺寸。由于舍弃部分下采样操作以节省参数会导致训练过程不稳定、收敛效率低，我们引入了即插即用模块时序光照单元（Temporal Light Unit, TLU），以实现更稳定的训练和更优的性能。TLU将时间步与去噪后的图像特征关联，建立去噪步骤的时序依赖关系，提升去噪效果。此外，我们注意到使用扩散模型恢复图像时可能出现光谱偏移问题，为此进一步引入色度平衡器（Chroma Balancer, CB）来缓解该问题。我们的LighTDiff在计算效率上表现优异，性能优于多种主流的低光照图像增强（LLIE）方法。
英文摘要：
Advances in endoscopy use in surgeries face challenges like inadequate lighting. Deep learning, notably the Denoising Diffusion Probabilistic Model (DDPM), holds promise for low-light image enhancement in the medical field. However, DDPMs are computationally demanding and slow, limiting their practical medical applications. To bridge this gap, we propose a lightweight DDPM, dubbed LighTDiff. It adopts a T-shape model architecture to capture global structural information using low-resolution images and gradually recover the details in subsequent denoising steps. We further prone the model to significantly reduce the model size while retaining performance. While discarding certain downsampling operations to save parameters leads to instability and low efficiency in convergence during the training, we introduce a Temporal Light Unit (TLU), a plug-and-play module, for more stable training and better performance. TLU associates time steps with denoised image features, establishing temporal dependencies of the denoising steps and improving denoising outcomes. Moreover, while recovering images using the diffusion model, potential spectral shifts were noted. We further introduce a Chroma Balancer (CB) to mitigate this issue. Our LighTDiff outperforms many competitive LLIE methods with exceptional computational efficiency.
Benchmarking Endoscopic Surgical Image Restoration and Beyond
The Chinese University of Hong Kong（香港中文大学）；Dalian University of Technology（大连理工大学）；Southern Medical University（南方医科大学）；Xidian University（西安电子科技大学）；Wuhan University（武汉大学）
Imperial College London（帝国理工学院）；University of Cambridge（剑桥大学）；PAII Inc（PAII公司）
In Robot-Assisted Minimally Invasive Surgery (RAMIS), a camera assistant is normally required to control the position and zooming ratio of the laparoscope, following the surgeon's instructions. However, moving the laparoscope frequently may lead to unstable and suboptimal views, while the adjustment of zooming ratio may interrupt the workflow of the surgical operation. To this end, we propose a multi-scale Generative Adversarial Network (GAN)-based video super-resolution method to construct a framework for automatic zooming ratio adjustment. It can provide automatic real-time zooming for high-…
Accurate and robust tracking and reconstruction of the surgical scene is a critical enabling technology toward autonomous robotic surgery. Existing algorithms for 3D perception in surgery mainly rely on geometric information, while we propose to also leverage semantic information inferred from the endoscopic video using image segmentation algorithms. In this paper, we present a novel, comprehensive surgical perception framework, Semantic-SuPer, that integrates geometric and semantic information to facilitate data association, 3D reconstruction, and tracking of endoscopic scenes, benefiting dow…
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information…
SurgVisAgent: Multimodal Agentic Model for Versatile Surgical Visual Enhancement
University of Chinese Academy of Sciences Multimedia Department, Xiaomi Inc Chinese Academy of Sciences, Institute of Automation Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences
Surgical robots are used to perform minimally invasive surgery and alleviate much of the burden imposed on surgeons. Our group has developed a surgical robot to aid in the removal of tumors at the base of the skull via access through the nostrils. To avoid injuring the patients, a collision-avoidance algorithm that depends on having an accurate model for the poses of the instruments' shafts is used. Given that the model's parameters can change over time owing to interactions between instruments and other disturbances, the online estimation of the poses of the instrument's shaft is essential. I…
Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London（伦敦大学学院威康/EPSRC介入与外科科学中心）；Department of Neurosurgery, The National Hospital for Neurology and Neurosurgery（国立神经病学与神经外科医院神经外科）
Video motion magnification could directly highlight subsurface blood vessels in endoscopic video in order to prevent inadvertent damage and bleeding. Applying motion filters to the full surgical image is however sensitive to residual motion from the surgical instruments and can impede practical application due to aberration motion artefacts. By storing the temporal filter response from local spatial frequency information for a single cardiovascular cycle prior to tool introduction to the scene, a filter can be used to determine if motion magnification should be active for a spatial region of t…
Robust Surgical Tools Detection in Endoscopic Videos with Noisy Data
Information Technology University（信息技术大学）；University of Bristol（布里斯托尔大学）；Bristol Heart Institute（布里斯托尔心脏研究所）；University of the West of England（西英格兰大学）；Qatar University（卡塔尔大学）
Mannheim University of Applied Sciences（曼海姆应用科学大学）；Magdeburg University（马格德堡大学）；Heidelberg University Hospital（海德堡大学医院）
Phantoms for surgical training are able to mimic cutting and suturing properties and patient-individual shape of organs, but lack a realistic visual appearance that captures the heterogeneity of surgical scenes. In order to overcome this in endoscopic approaches, hyperrealistic concepts have been proposed to be used in an augmented reality-setting, which are based on deep image-to-image transformation methods. Such concepts are able to generate realistic representations of phantoms learned from real intraoperative endoscopic sequences. Conditioned on frames from the surgical training process, …
Endoscopic images typically contain several artifacts. The artifacts significantly impact image analysis result in computer-aided diagnosis. Convolutional neural networks (CNNs), a type of deep learning, can removes such artifacts. Various architectures have been proposed for the CNNs, and the accuracy of artifact removal varies depending on the choice of architecture. Therefore, it is necessary to determine the artifact removal accuracy, depending on the selected architecture. In this study, we focus on endoscopic surgical instruments as artifacts, and determine and discuss the artifact remov…
CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Technical University Munich（慕尼黑工业大学）；Riwolink GmbH（Riwolink有限公司）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Shanghai Jiao Tong University（上海交通大学）；IPCA（葡萄牙卡瓦多和阿韦河理工学院）；University of Minho（米尼奥大学）
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Suzhou Institute of Intelligent Computing Technology, Chinese Academy of Sciences（中国科学院苏州智能计算技术研究院）
Robotic-assisted surgery allows surgeons to conduct precise surgical operations with stereo vision and flexible motor control. However, the lack of 3D spatial perception limits situational awareness during procedures and hinders mastering surgical skills in the narrow abdominal space. Depth estimation, as a representative perception task, is typically defined as an image reconstruction problem. In this work, we show that depth estimation can be reformed from a 3D surface perspective. We propose a loss function for depth estimation that integrates the surface-aware constraints, leading to a fas…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）
Image segmentation has been increasingly applied in medical settings as recent developments have skyrocketed the potential applications of deep learning. Urology, specifically, is one field of medicine that is primed for the adoption of a real-time image segmentation system with the long-term aim of automating endoscopic stone treatment. In this project, we explored supervised deep learning models to annotate kidney stones in surgical endoscopic video feeds. In this paper, we describe how we built a dataset from the raw videos and how we developed a pipeline to automate as much of the process …
EndoControlMag: Robust Endoscopic Vascular Motion Magnification with Periodic Reference Resetting and Hierarchical Tissue-aware Dual-Mask Control
organization= Department of Electronic Engineering, The Chinese University of Hong Kong , state= Hong Kong SAR , country= China；organization= The Chinese University of Hong Kong Shenzhen Research Institute , city= Shen Zhen , state= Guangdong , country= China；organization= The First Affiliated Hospital of Wenzhou Medical University , city= Wenzhou , state= Zhejiang , country= China；organization= Department of General Surgery \& Guangdong Provincial Key Laboratory of Precision Medicine for Gastrointestinal Tumor, Nanfang Hospital, Southern Medical University , city= Guang Zhou , state= Guangdong , country= China；organization= Department of Otolaryngology-Head；Neck Surgery, Singapore General Hospital, Duke-NUS Medical School , country= Singapore；organization= Shenzhen Key Laboratory of Robotics Perception；the Department of Electronic；Electrical Engineering, Southern University of Science
F2PASeg: Feature Fusion for Pituitary Anatomy Segmentation in Endoscopic Surgery
1Centre for Artificial Intelligence；Robotics, Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences 2The University of Hong Kong 3Peking Union Medical College Hospital, Chinese Academy of Medical Science
SurgiSR4K: A High-Resolution Endoscopic Video Dataset for Robotic-Assisted Minimally Invasive Procedures
Intuitive Surgical, Inc.（Intuitive Surgical公司）；Johns Hopkins Medicine Neurosurgery（约翰霍普金斯医学神经外科）；Johns Hopkins University Electrical and Computer Engineering（约翰霍普金斯大学电气与计算机工程）；University of British Columbia Electrical and Computer Engineering（不列颠哥伦比亚大学电气与计算机工程）；Wilford & Kate Bailey Small Animal Teaching Hospital（威尔福德与凯蒂·贝利小动物教学医院）
Rochester Institute of Technology（罗切斯特理工学院）；University of Washington（华盛顿大学）；Seattle Children’s Hospital（西雅图儿童医院）
Abstract. Purpose: This paper presents a scheme for generating virtual intraoperative CT scans in order to improve surgical completeness in Endoscopic Sinus Surgeries (ESS). Approach: The work presents three methods, the tip motion-based, the tip trajectory-based, and the instrument based, along with non-parametric smoothing and Gaussian Process Regression, for virtual intraoperative CT generation. Results: The proposed methods studied and compared on ESS performed on cadavers. Surgical results show all three methods improve the Dice Similarity Coefficients > 86%, with F-score > 92% and precis…
National University of Singapore（新加坡国立大学）；Imperial College London（帝国理工学院）；National Institute of Technology, Tiruchirappalli（蒂鲁吉拉伯利国家理工学院）；Chinese University of Hong Kong（香港中文大学）；Shun Hing Institute of Advanced Engineering（信兴高等工程研究所）
Visual question answering (VQA) in surgery is largely unexplored. Expert surgeons are scarce and are often overloaded with clinical and academic workloads. This overload often limits their time answering questionnaires from patients, medical students or junior residents related to surgical procedures. At times, students and junior residents also refrain from asking too many questions during classes to reduce disruption. While computer-aided simulators and recording of past surgical procedures have been made available for them to observe and improve their skills, they still hugely rely on medic…
