目的：手术场景理解在内窥镜手术中未来干预辅助系统的技术栈中起着关键作用。为此，跟踪内窥镜位姿是一个关键组成部分，但由于光照条件、组织变形和器官的呼吸运动，仍然具有挑战性。方法：我们提出了一种针对立体内窥镜的解决方案，通过估计深度和光流来最小化两个用于相机位姿估计的几何损失。最重要的是，我们引入了两个学习到的自适应逐像素权重映射，根据输入图像内容平衡贡献。为此，我们训练了一个 Deep Declarative Network，以利用深度学习的表达能力和一种新颖的基于几何的优化方法的鲁棒性。我们在公开可用的 SCARED 数据集上验证了我们的方法，并引入了一个新的体内数据集 StereoMIS，该数据集包含更广泛的典型观察到的手术设置。结果：我们的方法在平均情况下优于最先进的方法，更重要的是，在组织变形和呼吸运动可见的困难场景中也是如此。我们观察到，我们提出的权重映射减弱了图像中模糊区域（如变形组织）像素的贡献。结论：我们证明了我们的解决方案在具有挑战性的内窥镜手术场景中稳健估计相机位姿的有效性。我们的贡献可用于改进相关任务，如同步定位与建图（SLAM）或 3D 重建，从而推进微创手术中的手术场景理解。
英文摘要：
Purpose: Surgical scene understanding plays a critical role in the technology stack of tomorrow's intervention-assisting systems in endoscopic surgeries. For this, tracking the endoscope pose is a key component, but remains challenging due to illumination conditions, deforming tissues and the breathing motion of organs. Method: We propose a solution for stereo endoscopes that estimates depth and optical flow to minimize two geometric losses for camera pose estimation. Most importantly, we introduce two learned adaptive per-pixel weight mappings that balance contributions according to the input image content. To do so, we train a Deep Declarative Network to take advantage of the expressiveness of deep-learning and the robustness of a novel geometric-based optimization approach. We validate our approach on the publicly available SCARED dataset and introduce a new in-vivo dataset, StereoMIS, which includes a wider spectrum of typically observed surgical settings. Results: Our method outperforms state-of-the-art methods on average and more importantly, in difficult scenarios where tissue deformations and breathing motion are visible. We observed that our proposed weight mappings attenuate the contribution of pixels on ambiguous regions of the images, such as deforming tissues. Conclusion: We demonstrate the effectiveness of our solution to robustly estimate the camera pose in challenging endoscopic surgical scenes. Our contributions can be used to improve related tasks like simultaneous localization and mapping (SLAM) or 3D reconstruction, therefore advancing surgical scene understanding in minimally-invasive surgery.
ExtraGS: Enhancing Endoscopic View Extrapolation via Diffusion-Guided 3D Gaussian Splatting
The School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, and the Shanghai Key Laboratory of Navigation and Location-Based Services（上海交通大学自动化与智能感知学院以及上海市导航与位置服务重点实验室）；Global College, Shanghai Jiao Tong University（上海交通大学密西根学院）；Shanghai Ninth People’s Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属第九人民医院）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zurich（苏黎世联邦理工学院）；Middle East Technical University（中东技术大学）；Sabanci University（萨班哲大学）；University of Coimbra（科英布拉大学）
Since the development of capsule endoscopcy technology, substantial progress were made in converting passive capsule endoscopes to robotic active capsule endoscopes which can be controlled by the doctor. However, robotic capsule endoscopy still has some challenges. In particular, the use of such devices to generate a precise and globally consistent three-dimensional (3D) map of the entire inner organ remains an unsolved problem. Such global 3D maps of inner organs would help doctors to detect the location and size of diseased areas more accurately, precisely, and intuitively, thus permitting m…
Endo3R: Unified Online Reconstruction from Dynamic Monocular Endoscopic Video
The Chinese University of Hong Kong（香港中文大学）；Hong Kong Center for Logistics Robotics（香港物流机器人中心）；Johns Hopkins University（约翰斯·霍普金斯大学）；Shanghai Jiao Tong University（上海交通大学）
4D Monocular Surgical Reconstruction under Arbitrary Camera Motions
Department of Mechanical and Automation Engineering（机械与自动化工程系）；T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong（T Stone机器人研究院，香港中文大学，香港）；Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China（沈阳自动化研究所，中国科学院，沈阳，中国）；State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China（机器人与智能系统国家重点实验室，中国，沈阳）；School of Integrated Circuits, Shanghai Jiao Tong University, Shanghai, China（集成电路学院，上海交通大学，中国，上海）；School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China（自动化与智能感知学院，上海交通大学，中国，上海）；Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China（系统控制与信息处理重点实验室，中国教育部，中国，上海）
EndoFlow-SLAM: Real-Time Endoscopic SLAM with Flow-Constrained Gaussian Splatting
School of Advanced Technology, Xi'an Jiaotong Liverpool University（西安交通大学利物浦大学先进技术学院）；School of AI and Advanced Computing, Xi'an Jiaotong Liverpool University（西安交通大学利物浦大学人工智能与高级计算学院）；School of Physical Sciences, University of Liverpool（利物浦大学物理科学学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
MedGS: Gaussian Splatting for Multi-Modal 3D Medical Imaging
Jagiellonian University, Poland（雅盖隆大学，波兰）；Warsaw University of Technology, Poland（华沙理工大学，波兰）；IDEAS Research Institute, Poland（IDEAS研究所，波兰）；Military Institute of Medicine, Poland（军事医学研究所，波兰）
In the gastrointestinal (GI) tract endoscopy field, ingestible wireless capsule endoscopy is emerging as a novel, minimally invasive diagnostic technology for inspection of the GI tract and diagnosis of a wide range of diseases and pathologies. Since the development of this technology, medical device companies and many research groups have made substantial progress in converting passive capsule endoscopes to robotic active capsule endoscopes with most of the functionality of current active flexible endoscopes. However, robotic capsule endoscopy still has some challenges. In particular, the use…
ColonSplat: Reconstruction of Peristaltic Motion in Colonoscopy with Dynamic Gaussian Splatting
Jagiellonian University, Kraków, Poland（杰洛尼摩大学）；Warsaw University of Technology, Poland（华沙技术大学）；Sano Centre for Computational Medicine, Kraków, Poland（Sano计算医学中心）；University of Verona, Italy（威尼斯大学）；IDEAS Research Institute, Warsaw, Poland（IDEAS研究机构）
University of Toronto（多伦多大学）；Carnegie Mellon University（卡内基梅隆大学）；Simon Fraser University（西蒙弗雷泽大学）；Google（谷歌）；Vector Institute（向量研究所）
Camera pose estimation is a key step in standard 3D reconstruction pipelines that operate on a dense set of images of a single object or scene. However, methods for pose estimation often fail when only a few images are available because they rely on the ability to robustly identify and match visual features between image pairs. While these methods can work robustly with dense camera views, capturing a large set of images can be time-consuming or impractical. We propose SparsePose for recovering accurate camera poses given a sparse set of wide-baseline images (fewer than 10). The method learns …
EndoMD-SLAM: Endoscopic Gaussian Splatting SLAM under Optical Degradation with Memory and Static-Transient Decomposition
Texas A&M University（德克萨斯农工大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Minnesota（明尼苏达大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy
Sano Centre for Computional Medicine, Poland（波兰萨诺计算医学中心）；Warsaw University of Technology, Poland（波兰华沙理工大学）；Doctoral School of Exact and Natural Sciences, Jagiellonian University, Poland（杰齐尔大学精确与自然科学博士学院）；Faculty of Mathematics and Computer Science, Jagiellonian University, Poland（杰齐尔大学数学与计算机科学学院）；University of Verona, Italy（意大利威尼斯大学）；IDEAS Research Institute, Poland（波兰IDEAS研究所）
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Suzhou Institute of Intelligent Computing Technology, Chinese Academy of Sciences（中国科学院苏州智能计算技术研究院）
Robotic-assisted surgery allows surgeons to conduct precise surgical operations with stereo vision and flexible motor control. However, the lack of 3D spatial perception limits situational awareness during procedures and hinders mastering surgical skills in the narrow abdominal space. Depth estimation, as a representative perception task, is typically defined as an image reconstruction problem. In this work, we show that depth estimation can be reformed from a 3D surface perspective. We propose a loss function for depth estimation that integrates the surface-aware constraints, leading to a fas…
Institute of Biomedical Engineering, Bogazici University（波加兹奇大学生物医学工程研究所）；Computer Science Department, University of Oxford（牛津大学计算机科学系）；Brigham and Women’s Hospital, Harvard Medical School（哈佛医学院布莱根妇女医院）；Cancer Data Science, Dana Farber Cancer Institute（丹娜-法伯癌症研究所癌症数据科学部）；Cancer Program, Broad Institute of Harvard and MIT（哈佛-麻省理工学院布罗德研究所癌症项目）；Institute for Systems and Robotics, University of Coimbra（科英布拉大学系统与机器人研究所）；Faculty of Medicine, Clinical Academic Center of Coimbra, University of Coimbra（科英布拉大学科英布拉临床学术中心医学院）；Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物医学工程系）
Deep learning techniques hold promise to develop dense topography reconstruction and pose estimation methods for endoscopic videos. However, currently available datasets do not support effective quantitative benchmarking. In this paper, we introduce a comprehensive endoscopic SLAM dataset consisting of 3D point cloud data for six porcine organs, capsule and standard endoscopy recordings as well as synthetically generated data. A Panda robotic arm, two commercially available capsule endoscopes, two conventional endoscopes with different camera properties, and two high precision 3D scanners were…
The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Reconstructing the scene of robotic surgery from the stereo endoscopic video is an important and promising topic in surgical data science, which potentially supports many applications such as surgical visual perception, robotic surgery education and intra-operative context awareness. However, current methods are mostly restricted to reconstructing static anatomy assuming no tissue deformation, tool occlusion and de-occlusion, and camera movement. However, these assumptions are not always satisfied in minimal invasive robotic surgeries. In this work, we present an efficient reconstruction pipel…
We present a new algorithm for single camera 3D reconstruction, or 3D input for human-computer interfaces, based on precise tracking of an elongated object, such as a pen, having a pattern of colored bands. To configure the system, the user provides no more than one labelled image of a handmade pointer, measurements of its colored bands, and the camera's pinhole projection matrix. Other systems are of much higher cost and complexity, requiring combinations of multiple cameras, stereocameras, and pointers with sensors and lights. Instead of relying on information from multiple devices, we exami…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；FZI Research Center for Information Technology（FZI信息技术研究中心）
In this paper, we present TEScalib, a novel extrinsic self-calibration approach of LiDAR and stereo camera using the geometric and photometric information of surrounding environments without any calibration targets for automated driving vehicles. Since LiDAR and stereo camera are widely used for sensor data fusion on automated driving vehicles, their extrinsic calibration is highly important. However, most of the LiDAR and stereo camera calibration approaches are mainly target-based and therefore time consuming. Even the newly developed targetless approaches in last years are either inaccurate…
Center for Robotics, University of Bonn（博恩大学机器人中心）；MAVLab, TU Delft（代尔夫特理工大学MAVLab）；Lamarr Institute for Machine Learning and Artificial Intelligence（兰玛尔人工智能与机器学习研究所）
Bournemouth University（伯恩茅斯大学）；University of Chester（切斯特大学）；University of Bradford（布拉德福德大学）
One of the major challenges in Minimally Invasive Surgery (MIS) such as laparoscopy is the lack of depth perception. In recent years, laparoscopic scene tracking and surface reconstruction has been a focus of investigation to provide rich additional information to aid the surgical process and compensate for the depth perception issue. However, robust 3D surface reconstruction and augmented reality with depth perception on the reconstructed scene are yet to be reported. This paper presents our work in this area. First, we adopt a state-of-the-art visual simultaneous localization and mapping (SL…
Estimating a scene reconstruction and the camera motion from in-body videos is challenging due to several factors, e.g. the deformation of in-body cavities or the lack of texture. In this paper we present Endo-Depth-and-Motion, a pipeline that estimates the 6-degrees-of-freedom camera pose and dense 3D scene models from monocular endoscopic videos. Our approach leverages recent advances in self-supervised depth networks to generate pseudo-RGBD frames, then tracks the camera pose using photometric residuals and fuses the registered depth maps in a volumetric representation. We present an extens…
SenseTime Research（商汤科技研究院）；The University of Hong Kong（香港大学）
Precise localization of polyp is crucial for early cancer screening in gastrointestinal endoscopy. Videos given by endoscopy bring both richer contextual information as well as more challenges than still images. The camera-moving situation, instead of the common camera-fixed-object-moving one, leads to significant background variation between frames. Severe internal artifacts (e.g. water flow in the human body, specular reflection by tissues) can make the quality of adjacent frames vary considerately. These factors hinder a video-based model to effectively aggregate features from neighborhood …
College of Electrical and Electronic Engineering, Shandong University of Technology（山东理工大学电气与电子工程学院）
Active stereo vision is important in reconstructing objects without obvious textures. However, it is still very challenging to extract and match the projected patterns from two camera views automatically and robustly. In this paper, we propose a new pattern extraction method and a new stereo vision matching method based on our novel structured light pattern. Instead of using the widely used 2D disparity to calculate the depths of the objects, we use the ray intersection to compute the 3D shapes directly. Experimental results showed that the proposed approach could reconstruct the 3D shape of t…
Tel-Aviv University（特拉维夫大学）；AICFVE Beijing Film Academy（北京电影学院AICFVE）；Peking University（北京大学）；Shandong University（山东大学）；Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Analyzing human motion is a challenging task with a wide variety of applications in computer vision and in graphics. One such application, of particular importance in computer animation, is the retargeting of motion from one performer to another. While humans move in three dimensions, the vast majority of human motions are captured using video, requiring 2D-to-3D pose and camera recovery, before existing retargeting approaches may be applied. In this paper, we present a new method for retargeting video-captured motion between different human performers, without the need to explicitly reconstru…
The Dresden Dataset for 4D Reconstruction of Non-Rigid Abdominal Surgical Scenes
Department of Translational Surgical Oncology, National Center for Tumor Diseases (NCT), NCT/UCC Dresden, a partnership between DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology, and Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany（转化外科肿瘤中心部门，肿瘤疾病国家中心（NCT），NCT/UCC德累斯顿，由DKFZ、医学院和卡尔·戈斯瓦尔德·卡尔医院、德累斯顿技术大学以及德累斯顿-罗斯恩多夫赫尔姆霍兹中心（HZDR）组成的合作机构，德累斯顿，德国）；Department of Translational Surgical Oncology, NCT/UCC Dresden, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（转化外科肿瘤中心部门，NCT/UCC德累斯顿，医学院和卡尔·戈斯瓦尔德·卡尔医院，德累斯顿技术大学）；Department for Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden（visceral、胸腔和血管外科部门，医学院和卡尔·戈斯瓦尔德·卡尔医院，德累斯顿技术大学）；Centre for Tactile Internet with Human-in-the-Loop (CeTI), Technical University Dresden（人机交互触觉互联网中心（CeTI），德累斯顿技术大学）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（计算机科学与工程部门，香港中文大学）
Australian National University（澳大利亚国立大学）；Data61 – CSIRO（澳大利亚联邦科学与工业研究组织 Data61）
Estimating the 6-DoF pose of a camera from a single image relative to a pre-computed 3D point-set is an important task for many computer vision applications. Perspective-n-Point (PnP) solvers are routinely used for camera pose estimation, provided that a good quality set of 2D-3D feature correspondences are known beforehand. However, finding optimal correspondences between 2D key-points and a 3D point-set is non-trivial, especially when only geometric (position) information is known. Existing approaches to the simultaneous pose and correspondence problem use local optimisation, and are therefo…
NRGS-SLAM: Monocular Non-Rigid SLAM for Endoscopy via Deformation-Aware 3D Gaussian Splatting
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong（机械与自动化工程系和T Stone机器人研究所，香港中文大学）；School of Automation and Intelligent Sensing, the State Key Laboratory of Avionics Integration and Aviation System of-Systems Synthesis, and Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University（自动化与智能感知学院、航空系统集成与航空系统综合国家重点实验室、上海导航与定位服务重点实验室，上海交通大学）
Saarland University（萨尔兰大学）；University of Ottawa（渥太华大学）；National Research Council of Canada（加拿大国家研究委员会）
We present an approach to robustly track the geometry of an object that deforms over time from a set of input point clouds captured from a single viewpoint. The deformations we consider are caused by applying forces to known locations on the object's surface. Our method combines the use of prior information on the geometry of the object modeled by a smooth template and the use of a linear finite element method to predict the deformation. This allows the accurate reconstruction of both the observed and the unobserved sides of the object. We present tracking results for noisy low-quality point c…
Video streams are utilised to guide minimally-invasive surgery and diagnostic procedures in a wide range of procedures, and many computer assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modeling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from…
State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
In this paper, we present RKD-SLAM, a robust keyframe-based dense SLAM approach for an RGB-D camera that can robustly handle fast motion and dense loop closure, and run without time limitation in a moderate size scene. It not only can be used to scan high-quality 3D models, but also can satisfy the demand of VR and AR applications. First, we combine color and depth information to construct a very fast keyframe-based tracking method on a CPU, which can work robustly in challenging cases (e.g.~fast camera motion and complex loops). For reducing accumulation error, we also introduce a very effici…
Camera calibration for the surround-view system: a benchmark and dataset
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Beijing Key Laboratory of Advanced Information Science and Network（北京先进信息科学与网络重点实验室）
