Automatic Calibration of a Multi-Camera System with Limited Overlapping Fields of View for 3D Surgical Scene Reconstruction
Research in Orthopedic Computer Science（骨科计算机科学研究）；University Hospital Balgrist（巴尔格斯特大学医院）；University of Zurich（苏黎世大学）；Computer Vision and Geometry（计算机视觉与几何）；ETH Zurich（苏黎世联邦理工学院）
Tim Flückiger, Jonas Hein, Valery Fischer, Philipp Fürnstahl, Lilian Calvet
The purpose of this study is to develop an automated and accurate external camera calibration method for multi-camera systems used in 3D surgical scene reconstruction (3D-SSR), eliminating the need for operator intervention or specialized expertise. The method specifically addresses the problem of limited overlapping fields of view caused by significant variations in optical zoom levels and camera locations. We contribute a novel, fast, and fully automatic calibration method based on the projection of multi-scale markers (MSMs) using a ceiling-mounted projector. MSMs consist of 2D patterns projected at varying scales, ensuring accurate extraction of well distributed point correspondences across significantly different viewpoints and zoom levels. Validation is performed using both synthetic and real data captured in a mock-up OR, with comparisons to traditional manual marker-based methods as well as markerless calibration methods. The method achieves accuracy comparable to manual, operator-dependent calibration methods while exhibiting higher robustness under conditions of significant differences in zoom levels. Additionally, we show that state-of-the-art Structure-from-Motion (SfM) pipelines are ineffective in 3D-SSR settings, even when additional texture is projected onto the OR floor. The use of a ceiling-mounted entry-level projector proves to be an effective alternative to operator-dependent, traditional marker-based methods, paving the way for fully automated 3D-SSR.
Autonomy is the frontier of research in robotic surgery and its aim is to improve the quality of surgical procedures in the next future. One fundamental requirement for autonomy is advanced perception capability through vision sensors. In this paper, we propose a novel calibration technique for a surgical scenario with da Vinci robot. Calibration of the camera and the robot is necessary for precise positioning of the tools in order to emulate the high performance surgeons. Our calibration technique is tailored for RGB-D camera. Different tests performed on relevant use cases for surgery prove …
We propose an easy-to-use non-overlapping camera calibration method. First, successive images are fed to a PoseNet-based network to obtain ego-motion of cameras between frames. Next, the pose between cameras are estimated. Instead of using a batch method, we propose an on-line method of the inter-camera pose estimation. Furthermore, we implement the entire procedure on a computation graph. Experiments with simulations and the KITTI dataset show the proposed method to be effective in simulation.
SurgNavAR: An Augmented Reality Surgical Navigation Framework for Optical See-Through Head Mounted Displays
Biomedical Imaging Group Rotterdam, Department of Radiology & Nuclear Medicine, Erasmus MC（鹿特丹伊拉斯姆斯大学医学中心放射与核医学系鹿特丹生物医学成像组）；Department of Oral and Maxillofacial Surgery, Erasmus MC（鹿特丹伊拉斯姆斯大学医学中心口腔颌面外科）；University of Groningen（格罗宁根大学）；Department of High Performance Compute & Visualization, SURF bv（SURF bv高性能计算与可视化部）；Department of Neurosurgery, Erasmus MC（鹿特丹伊拉斯姆斯大学医学中心神经外科）；Trauma Research Unit, Department of Surgery, Erasmus MC（鹿特丹伊拉斯姆斯大学医学中心外科创伤研究单元）
National University of Singapore（新加坡国立大学）；NUSRI Suzhou（新加坡国立大学苏州研究院）；Singapore General Hospital（新加坡中央医院）；The Chinese University of Hong Kong（香港中文大学）；Imperial College London（伦敦帝国理工学院）
Generating surgical reports aimed at surgical scene understanding in robot-assisted surgery can contribute to documenting entry tasks and post-operative analysis. Despite the impressive outcome, the deep learning model degrades the performance when applied to different domains encountering domain shifts. In addition, there are new instruments and variations in surgical tissues appeared in robotic surgery. In this work, we propose class-incremental domain adaptation (CIDA) with a multi-layer transformer-based model to tackle the new classes and domain shift in the target domain to generate surg…
Multi-camera calibration with pattern rigs, including for non-overlapping cameras: CALICO
USDA-ARS-AFRS（美国农业部农业研究服务局农业和食品研究处）；University of Florida（佛罗里达大学）；University of California-Davis（加州大学戴维斯分校）；Embrapa Digital Agriculture（巴西农业研究公司数字农业研究所）
SurgCalib: Gaussian Splatting-Based Hand-Eye Calibration for Robot-Assisted Minimally Invasive Surgery
Robotics and Control Laboratory, University of British Columbia（机器人与控制实验室，不列颠哥伦比亚大学）；Department of Biomedical Engineering, National University of Singapore（生物医学工程系，新加坡国立大学）；Division of Otolaryngology – Head and Neck Surgery, University of British Columbia（耳鼻喉科–头颈外科部，不列颠哥伦比亚大学）
This paper presents a framework for the targetless extrinsic calibration of stereo cameras and Light Detection and Ranging (LiDAR) sensors with a non-overlapping Field of View (FOV). In order to solve the extrinsic calibrations problem under such challenging configuration, the proposed solution exploits road markings as static and robust features among the various dynamic objects that are present in urban environment. First, this study utilizes road markings that are commonly captured by the two sensor modalities to select informative images for estimating the extrinsic parameters. In order to…
In percutaneous orthopedic interventions the surgeon attempts to reduce and fixate fractures in bony structures. The complexity of these interventions arises when the surgeon performs the challenging task of navigating surgical tools percutaneously only under the guidance of 2D interventional X-ray imaging. Moreover, the intra-operatively acquired data is only visualized indirectly on external displays. In this work, we propose a flexible Augmented Reality (AR) paradigm using optical see-through head mounted displays. The key technical contribution of this work includes the marker-less and dyn…
Multi-view Video-Pose Pretraining for Operating Room Surgical Activity Recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Intuitive Surgical Inc.（直觉外科公司）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Sensor calibration usually is a time consuming yet important task. While classical approaches are sensor-specific and often need calibration targets as well as a widely overlapping field of view (FOV), within this work, a cooperative intelligent vehicle is used as callibration target. The vehicleis detected in the sensor frame and then matched with the information received from the cooperative awareness messagessend by the coperative intelligent vehicle. The presented algorithm is fully automated as well as sensor-independent, relying only on a very common set of assumptions. Due to the direct…
Hamlyn Centre for Robotic Surgery, Imperial College London（伦敦帝国理工学院哈姆林机器人手术中心）；Institute of Image Processing & Pattern Recognition, Shanghai Jiao Tong University（上海交通大学图像处理与模式识别研究所）
In robotic surgery, task automation and learning from demonstration combined with human supervision is an emerging trend for many new surgical robot platforms. One such task is automated anastomosis, which requires bimanual needle handling and suture detection. Due to the complexity of the surgical environment and varying patient anatomies, reliable suture detection is difficult, which is further complicated by occlusion and thread topologies. In this paper, we propose a multi-stage framework for suture thread detection based on deep learning. Fully convolutional neural networks are used to ob…
Johns Hopkins University（约翰斯·霍普金斯大学）；Georgia Institute of Technology（佐治亚理工学院）；Université de Montréal（蒙特利尔大学）；Intuitive Surgical Inc.（直观外科公司）
Purpose: We describe a 3D multi-view perception system for the da Vinci surgical system to enable Operating room (OR) scene understanding and context awareness. Methods: Our proposed system is comprised of four Time-of-Flight (ToF) cameras rigidly attached to strategic locations on the daVinci Xi patient side cart (PSC). The cameras are registered to the robot's kinematic chain by performing a one-time calibration routine and therefore, information from all cameras can be fused and represented in one common coordinate frame. Based on this architecture, a multi-view 3D scene semantic segmentati…
The SAGES Critical View of Safety Challenge: A Global Benchmark for AI-Assisted Surgical Quality Assessment
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家医学研究院）；ICube（ICube研究中心）；UMR7357（法国国家科研机构（UMR7357））；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University Hospital Cologne（科隆大学医院）；Global College（全球学院）；Shanghai Jiao Tong University（上海交通大学）；Chang Gung Memorial Hospital（长庚纪念医院）；A. Costa Hospital（A. Costa医院）；Maggiore Hospital - AUSL Bologna（马吉奥医院 - 玛格丽特医院）；Institute for Research against Digestive Cancer (IRCAD)（消化道癌症研究机构（IRCAD））；Lenox Hill Hospital（Lenox Hill医院）；Northwell Health（Northwell健康系统）；Albany Medical Center（阿尔巴尼医疗中心）；German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；Stanford University（斯坦福大学）；MultiModal Learning Lab（多模态学习实验室）；NAAMII；University of Aberdeen（阿伯丁大学）；Eindhoven University of Technology（埃因霍温理工大学）；UCL Hawkes Institute（UCL Hawkes研究所）；Dept of Computer Science, University College London（计算机科学系，伦敦大学学院）；University Health Network（大学健康网络）；Institute for Biomedical Informatics（生物医学信息研究所）；Rovira i Virgili University（罗维拉-维吉里大学）；Huazhong University of Science（华中科技大学）；Hefei University of Technology（合肥工业大学）；University of Pennsylvania（宾夕法尼亚大学）；Duke University（杜克大学）
Deep intra-operative illumination calibration of hyperspectral cameras
Siemens AG（西门子股份公司）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Heidelberg University Hospital（海德堡大学医院）；University Medical Center Mannheim（曼海姆大学医学中心）；DKFZ Hector Cancer Institute at the University Medical Center Mannheim（德国癌症研究中心赫克曼海姆大学医学中心癌症研究所）
Optimization-Based Calibration for Intravascular Ultrasound Volume Reconstruction
Inria；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家医学研究院）；ICube（ICube研究中心）；UMR7357（UMR7357研究单位）；University of Basel（巴塞尔大学）；Department of Biomedical Engineering（生物医学工程系）；Institute of Image-Guided Surgery（影像引导手术研究所）；IHU Strasbourg（斯特拉斯堡IHU）
Acquiring Submillimeter-Accurate Multi-Task Vision Datasets for Computer-Assisted Orthopedic Surgery
University Hospital Balgrist, University of Zurich（苏黎世大学医院巴尔格里斯分院、苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）；Institut de Recherche en Informatique de Toulouse（图卢兹计算机科学研究所）
Surgical scene understanding and multi-tasking learning are crucial for image-guided robotic surgery. Training a real-time robotic system for the detection and segmentation of high-resolution images provides a challenging problem with the limited computational resource. The perception drawn can be applied in effective real-time feedback, surgical skill assessment, and human-robot collaborative surgeries to enhance surgical outcomes. For this purpose, we develop a novel end-to-end trainable real-time Multi-Task Learning (MTL) model with weight-shared encoder and task-aware detection and segment…
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), UCL（伦敦大学学院威康/EPSRC介入与外科科学中心）；National Center for Tumor Diseases (NCT) Dresden（德累斯顿国家肿瘤疾病中心）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
In computer vision, reference datasets have been highly successful in promoting algorithmic development in stereo reconstruction. Surgical scenes gives rise to specific problems, including the lack of clear corner features, highly specular surfaces and the presence of blood and smoke. Publicly available datasets have been produced using CT and either phantom images or biological tissue samples covering a relatively small region of the endoscope field-of-view. We present a stereo-endoscopic reconstruction validation dataset based on CT (SERV-CT). Two {\it ex vivo} small porcine full torso cadav…
Endo3R: Unified Online Reconstruction from Dynamic Monocular Endoscopic Video
The Chinese University of Hong Kong（香港中文大学）；Hong Kong Center for Logistics Robotics（香港物流机器人中心）；Johns Hopkins University（约翰斯·霍普金斯大学）；Shanghai Jiao Tong University（上海交通大学）
TeleOR: Real-time Telemedicine System for Full-Scene Operating Room
School of Public Health, Zhejiang University（浙江大学公共卫生学院）；The Second Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院附属第二医院）；Institute of Wenzhou, Zhejiang University（浙江大学温州研究院）；School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Notre Dame（圣母大学）
Vellore Institute of Technology, Chennai（金奈韦洛尔理工大学）；Birla Institute of Technology and Science, Hyderabad（海得拉巴比尔拉理工学院）；Veermata Jijabai Technological Institute, Mumbai（孟买韦尔马塔·吉贾拜技术学院）；International Institute of Information and Technology, Hyderabad（海得拉巴国际信息技术学院）
With the advent of autonomous vehicles, LiDAR and cameras have become an indispensable combination of sensors. They both provide rich and complementary data which can be used by various algorithms and machine learning to sense and make vital inferences about the surroundings. We propose a novel pipeline and experimental setup to find accurate rigid-body transformation for extrinsically calibrating a LiDAR and a camera. The pipeling uses 3D-3D point correspondences in LiDAR and camera frame and gives a closed form solution. We further show the accuracy of the estimate by fusing point clouds fro…
Southwest Jiaotong University（西南交通大学）；The Chinese University of Hong Kong（香港中文大学）
Multiple Camera Systems (MCS) have been widely used in many vision applications and attracted much attention recently. There are two principle types of MCS, one is the Rigid Multiple Camera System (RMCS); the other is the Articulated Camera System (ACS). In a RMCS, the relative poses (relative 3-D position and orientation) between the cameras are invariant. While, in an ACS, the cameras are articulated through movable joints, the relative pose between them may change. Therefore, through calibration of an ACS we want to find not only the relative poses between the cameras but also the positions…
Department of Control Science & Engineering, the College of Electronic & Information Engineering, Tongji University（控制科学与工程系，电子与信息工程学院，同济大学）；School of Computer Science and Engineering, Central South University（计算机科学与工程学院，中南大学）；Beijing Institute of Aerospace Control Devices（北京航天控制器件研究所）；Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）
Hyperspectral imaging is one of the most promising techniques for intraoperative tissue characterisation. Snapshot mosaic cameras, which can capture hyperspectral data in a single exposure, have the potential to make a real-time hyperspectral imaging system for surgical decision-making possible. However, optimal exploitation of the captured data requires solving an ill-posed demosaicking problem and applying additional spectral corrections to recover spatial and spectral information of the image. In this work, we propose a deep learning-based image demosaicking algorithm for snapshot hyperspec…
Queensland University of Technology（昆士兰科技大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）；Stanford University（斯坦福大学）
This paper proposes the design of a custom mirror-based light field camera adapter that is cheap, simple in construction, and accessible. Mirrors of different shape and orientation reflect the scene into an upwards-facing camera to create an array of virtual cameras with overlapping field of view at specified depths, and deliver video frame rate light fields. We describe the design, construction, decoding and calibration processes of our mirror-based light field camera adapter in preparation for an open-source release to benefit the robotic vision community.
University of Pennsylvania School of Engineering and Applied Science（宾夕法尼亚大学工程与应用科学学院）
TagSLAM provides a convenient, flexible, and robust way of performing Simultaneous Localization and Mapping (SLAM) with AprilTag fiducial markers. By leveraging a few simple abstractions (bodies, tags, cameras), TagSLAM provides a front end to the GTSAM factor graph optimizer that makes it possible to rapidly design a range of experiments that are based on tags: full SLAM, extrinsic camera calibration with non-overlapping views, visual localization for ground truth, loop closure for odometry, pose estimation etc. We discuss in detail how TagSLAM initializes the factor graph in a robust way, an…
