Reconstructing deformable tissues from endoscopic stereo videos in robotic surgery is crucial for various clinical applications. However, existing methods relying only on implicit representations are computationally expensive and require dozens of hours, which limits further practical applications. To address this challenge, we introduce LerPlane, a novel method for fast and accurate reconstruction of surgical scenes under a single-viewpoint setting. LerPlane treats surgical procedures as 4D volumes and factorizes them into explicit 2D planes of static and dynamic fields, leading to a compact memory footprint and significantly accelerated optimization. The efficient factorization is accomplished by fusing features obtained through linear interpolation of each plane and enables using lightweight neural networks to model surgical scenes. Besides, LerPlane shares static fields, significantly reducing the workload of dynamic tissue modeling. We also propose a novel sample scheme to boost optimization and improve performance in regions with tool occlusion and large motions. Experiments on DaVinci robotic surgery videos demonstrate that LerPlane accelerates optimization by over 100$\times$ while maintaining high quality across various non-rigid deformations, showing significant promise for future intraoperative surgery applications.
Shanghai United Imaging Intelligence Co. Ltd（上海联影智能医疗科技有限公司）；Southern Medical University（南方医科大学）
In brain tumor diagnosis and surgical planning, segmentation of tumor regions and accurate analysis of surrounding normal tissues are necessary for physicians. Pathological variability often renders difficulty to register a well-labeled normal atlas to such images and to automatic segment/label surrounding normal brain tissues. In this paper, we propose a new registration approach that first segments brain tumor using a U-Net and then simulates missed normal tissues within the tumor region using a partial convolutional network. Then, a standard normal brain atlas image is registered onto such …
Reconstruction of the soft tissues in robotic surgery from endoscopic stereo videos is important for many applications such as intra-operative navigation and image-guided robotic surgery automation. Previous works on this task mainly rely on SLAM-based approaches, which struggle to handle complex surgical scenes. Inspired by recent progress in neural rendering, we present a novel framework for deformable tissue reconstruction from binocular captures in robotic surgery under the single-viewpoint setting. Our framework adopts dynamic neural radiance fields to represent deformable surgical scenes…
University of Technology, Sydney（悉尼科技大学）；Centre for Autonomous Systems（自主系统中心）
In minimal invasive surgery, it is important to rebuild and visualize the latest deformed shape of soft-tissue surfaces to mitigate tissue damages. This paper proposes an innovative Simultaneous Localization and Mapping (SLAM) algorithm for deformable dense reconstruction of surfaces using a sequence of images from a stereoscope. We introduce a warping field based on the Embedded Deformation (ED) nodes with 3D shapes recovered from consecutive pairs of stereo images. The warping field is estimated by deforming the last updated model to the current live model. Our SLAM system can: (1) Increment…
Towards Markerless Intraoperative Tracking of Deformable Spine Tissue
The Hamlyn Centre for Robotic Surgery, Imperial College London, London, UK, Department of Computing, Imperial College London, London, UK Department of Mechanical Engineering, Imperial College London, London, UK Department of Surgery \& Cancer, Imperial College London, London, UK Department of Neurosurgery, IRCCS Humanitas Research Hospital, Rozzano, Lombardy, Italy Department of Biomedical Sciences, Humanitas University, Pieve Emanuele, Milan, Italy. Stanford University School of Medicine, Stanford University, California, USA
GuidedMorph: Two-Stage Deformable Registration for Breast MRI
Department of Electrical and Computer Engineering, Duke University（电子工程系，杜克大学）；Department of Biostatistics & Bioinformatics, Duke University（生物统计学与生物信息学系，杜克大学）；Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida（健康结果与生物医学信息学系，医学院，佛罗里达大学）
SAGS: Self-Adaptive Alias-Free Gaussian Splatting for Dynamic Surgical Endoscopic Reconstruction
University of Technology Sydney, Australia（悉尼技术大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China（深圳先进技术研究所）；Shenyang Institute of Automation, Chinese Academy of Sciences, China（沈阳自动化研究所）
2D-3D Deformable Image Registration of Histology Slide and Micro-CT with ML-based Initialization
ImFusion GmbH（ImFusion有限公司）；Technical University of Munich（慕尼黑工业大学）；Institute of Pathology, Heinrich Heine University and University Hospital of Düsseldorf（海因里希·海涅大学与杜塞尔多夫大学医院病理研究所）；Fraunhofer Institute for High-Speed Dynamics, Ernst-Mach-Institut（弗劳恩霍夫高速动力学研究所（恩斯特·马赫研究所））
Aix Marseille Univ（艾克斯-马赛大学）；Université de Toulon（土伦大学）；CNRS（法国国家科学研究中心）
Dynamic MRI may capture temporal anatomical changes in soft tissue organs with high contrast but the obtained sequences usually suffer from limited volume coverage which makes the high resolution reconstruction of organ shape trajectories a major challenge in temporal studies. Because of the variability of abdominal organ shapes across time and subjects, the objective of this study is to go towards 3D dense velocity measurements to fully cover the entire surface and to extract meaningful features characterizing the observed organ deformations and enabling clinical action or decision. We presen…
Cherenkov Imaged Bio-morphological Features Verify Patient Positioning with Deformable Tissue Translocation in Breast Radiotherapy
Dartmouth College（达特茅斯学院）；Dartmouth Cancer Center（达特茅斯癌症中心）；Dartmouth Health（达特茅斯健康）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Missouri-Columbia（密苏里大学哥伦比亚分校）
SurgicalGaussian: Deformable 3D Gaussians for High-Fidelity Surgical Scene Reconstruction
Center for Digital Media Computing（数字媒体计算中心）；School of Film（电影学院）；School of Informatics（信息学院）；National Institute for Data Science in Health and Medicine（健康与医学数据科学国家研究院）；Xiamen University（厦门大学）；Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism（文化和旅游部闽台非物质文化遗产数字化保护与智能处理重点实验室）；Guangdong Provincial Key Laboratory IRADS（广东省IRADS重点实验室）；BNU-HKBU United International College（北京师范大学-香港浸会大学联合国际学院）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information…
University of California San Diego（加州大学圣迭戈分校）；East China Normal University（华东师范大学）
Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and re…
Ryerson University（瑞尔森大学）；University of Toronto（多伦多大学）；Institute for Biomedical Engineering, Science and Technology (iBEST)（生物医学工程、科学与技术研究所（iBEST））
We propose FlowReg, a deep learning-based framework for unsupervised image registration for neuroimaging applications. The system is composed of two architectures that are trained sequentially: FlowReg-A which affinely corrects for gross differences between moving and fixed volumes in 3D followed by FlowReg-O which performs pixel-wise deformations on a slice-by-slice basis for fine tuning in 2D. The affine network regresses the 3D affine matrix based on a correlation loss function that enforces global similarity. The deformable network operates on 2D image slices based on the optical flow netw…
STAR: A Fast and Robust Rigid Registration Framework for Serial Histopathological Images
School of Biological Science and Medical Engineering, Beihang University, Beijing, China（生物科学与医学工程学院，北航，北京，中国）；School of Computer Science and Engineering, Beihang University, Beijing, China（计算机科学与工程学院，北航，北京，中国）
Endo-NeRF++: Uncertainty-Aware Neural Rendering with Multi-Resolution Hash Encoding for Dynamic Surgical Scene Reconstruction
Finnish Center for Artificial Intelligence (FCAI)（芬兰人工智能中心）；ELLIS Unit Helsinki（欧洲学习与智能系统研究所赫尔辛基分部）；University of Helsinki（赫尔辛基大学）；Institute for Urban Studies (Urbaria)（城市研究所）；Institute of Sustainability Science (HELSUS)（可持续发展科学研究所）
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Boston Children’s Hospital（波士顿儿童医院）；University of Cape Town Faculty of Health Sciences（开普敦大学健康科学学院）；Wayne State University School of Medicine（韦恩州立大学医学院）；MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）
Skullstripping is defined as the task of segmenting brain tissue from a full head magnetic resonance image~(MRI). It is a critical component in neuroimage processing pipelines. Downstream deformable registration and whole brain segmentation performance is highly dependent on accurate skullstripping. Skullstripping is an especially challenging task for infant~(age range 0--18 months) head MRI images due to the significant size and shape variability of the head and the brain in that age range. Infant brain tissue development also changes the $T_1$-weighted image contrast over time, making consis…
Real-Time Spatio-Temporal Reconstruction of Dynamic Endoscopic Scenes with 4D Gaussian Splatting
University of Liverpool, Liverpool, UK（利物浦大学）；Xi'an Jiaotong-Liverpool University, Suzhou, China（西安交通大学利物浦大学）；University of Cambridge, Cambridge, UK（剑桥大学）
DeGenseGS: Geometrically and Semantically Decoupled Surgical Scene Understanding in 4D Gaussian Splatting
School of Automation, Southeast University（东南大学自动化学院）；School of Internet of Things, Nanjing University of Posts and Telecommunications（南京邮电大学物联网学院）；Department of Biomedical Engineering, National University of Singapore（新加坡国立大学生物医学工程系）
