Geometry OR Tracker: Universal Geometric Operating Room Tracking
Centre for Artificial Intelligence and Robotics（人工智能与机器人中心）；Hong Kong Institute of Science and Innovation Chinese Academy of Sciences（香港科学与创新研究院中国科学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)（多模态人工智能系统国家重点实验室）；Institute of Automation Chinese Academy of Sciences (CASIA)（自动化研究所中国科学院（CASIA））；Peking University（北京大学）；University of Trento, Trento, Italy（意大利特伦特大学）；Alibaba DAMO Academy（阿里达摩院）；Xiaomi Inc.（小米公司）；Technische Universität München（慕尼黑技术大学）
Yihua Shao, Kang Chen, Feng Xue, Siyu Chen, Long Bai, Hongyuan Yu, Hao Tang, Jinlin Wu, Nassir Navab
Geometry OR Tracker通过多视角几何校正和鲁棒3D点跟踪提升手术室多视角3D跟踪的几何一致性与准确性。
摘要：
在手术室（OR）中，世界级多视角3D跟踪支持下游应用，如外科医生行为识别，其中像距离和运动统计等物理有意义的量必须以米为单位进行测量。然而，真实的临床部署很少满足多视角融合和跟踪的几何前提：相机校准和RGB-D注册总是不可靠，导致跨视角几何不一致，从而在融合过程中产生'鬼影'现象，并降低共享OR坐标框架中的3D轨迹质量。为了解决这个问题，我们引入了Geometry OR Tracker，一个两阶段流程，首先通过多视角度量几何校正模块将不精确的校准校正为具有单个全局尺度的几何一致的相机设置，然后在统一的OR世界框架内执行遮挡鲁棒的3D点跟踪。在MM-OR基准上，改进的几何一致性带来了跟踪收益：我们的校正前端将跨视角深度不一致度降低到原始校准的30倍以上。消融研究进一步展示了校准质量和跟踪准确性之间的关系，表明改进的几何一致性会带来更强的世界框架跟踪。
英文摘要：
In operating rooms (OR), world-scale multi-view 3D tracking supports downstream applications such as surgeon behavior recognition, where physically meaningful quantities such as distances and motion statistics must be measured in meters. However, real clinical deployments rarely satisfy the geometric prerequisites for stable multi-view fusion and tracking: camera calibration and RGB-D registration are always unreliable, leading to cross-view geometric inconsistency that produces "ghosting" during fusion and degrades 3D trajectories in a shared OR coordinate frame. To address this, we introduce Geometry OR Tracker, a two-stage pipeline that first rectifies imprecise calibration into a scaleconsistent and geometrically consistent camera setup with a single global scale via a Multi-view Metric Geometry Rectification module, and then performs Occlusion-Robust 3D Point Tracking directly in the unified OR world frame. On the MM-OR benchmark, improved geometric consistency translates into tracking gains: our rectification front-end reduces cross-view depth disagreement by more than 30$\times$ compared to raw calibration. Ablation studies further demonstrate the relationship between calibration quality and tracking accuracy, showing that improved geometric consistency yields stronger world-frame tracking.
Automatic surgical activity recognition enables more intelligent surgical devices and a more efficient workflow. Integration of such technology in new operating rooms has the potential to improve care delivery to patients and decrease costs. Recent works have achieved a promising performance on surgical activity recognition; however, the lack of generalizability of these models is one of the critical barriers to the wide-scale adoption of this technology. In this work, we study the generalizability of surgical activity recognition models across operating rooms. We propose a new domain adaptati…
Data-driven approaches to assist operating room (OR) workflow analysis depend on large curated datasets that are time consuming and expensive to collect. On the other hand, we see a recent paradigm shift from supervised learning to self-supervised and/or unsupervised learning approaches that can learn representations from unlabeled datasets. In this paper, we leverage the unlabeled data captured in robotic surgery ORs and propose a novel way to fuse the multi-modal data for a single video frame or image. Instead of producing different augmentations (or 'views') of the same image or video frame…
TrackOR: Towards Personalized Intelligent Operating Rooms Through Robust Tracking
Computer Aided Medical Procedures, TU Munich, Germany（图宾根大学医学辅助技术中心）；Munich Center for Machine Learning, Germany（慕尼黑机器学习中心）；Minimally Invasive Surgery, University Hospital of Munich (LMU), Germany（慕尼黑大学微创手术中心）
ICube, University of Strasbourg, CNRS, IHU Strasbourg（斯特拉斯堡大学ICube实验室，法国国家科学研究中心，斯特拉斯堡大学医院）；Radiology Department, University Hospital of Strasbourg（斯特拉斯堡大学医院放射科）
Person detection and pose estimation is a key requirement to develop intelligent context-aware assistance systems. To foster the development of human pose estimation methods and their applications in the Operating Room (OR), we release the Multi-View Operating Room (MVOR) dataset, the first public dataset recorded during real clinical interventions. It consists of 732 synchronized multi-view frames recorded by three RGB-D cameras in a hybrid OR. It also includes the visual challenges present in such environments, such as occlusions and clutter. We provide camera calibration parameters, color a…
OR-VSKC: Resolving Visual-Semantic Knowledge Conflicts in Operating Rooms with Synthetic Data-Guided Alignment
Shanghai University of Engineering Science（上海工程技术大学）；Tencent YouTu Lab（腾讯优图实验室）；Clinical Research Unit, Zhongshan Hospital of Fudan University（复旦大学中山医院临床研究部）
Oxford Brookes University（牛津布鲁克斯大学）；Imperial College London（帝国理工学院）
Nowadays, there are more surgical procedures that are being performed using minimally invasive surgery (MIS). This is due to its many benefits, such as minimal post-operative problems, less bleeding, minor scarring, and a speedy recovery. However, the MIS's constrained field of view, small operating room, and indirect viewing of the operating scene could lead to surgical tools colliding and potentially harming human organs or tissues. Therefore, MIS problems can be considerably reduced, and surgical procedure accuracy and success rates can be increased by using an endoscopic video feed to dete…
Automatic recognition of surgical activities in the operating room (OR) is a key technology for creating next generation intelligent surgical devices and workflow monitoring/support systems. Such systems can potentially enhance efficiency in the OR, resulting in lower costs and improved care delivery to the patients. In this paper, we investigate automatic surgical activity recognition in robot-assisted operations. We collect the first large-scale dataset including 400 full-length multi-perspective videos from a variety of robotic surgery cases captured using Time-of-Flight cameras. We densely…
Surgical Re-enactment for Operating Room Workflow Datasets
TUM University Hospital（慕尼黑工业大学附属医院）；TUM School of Medicine and Health（慕尼黑工业大学医学与健康学院）；TUM School of Computation, Information, and Technology（慕尼黑工业大学计算、信息与技术学院）；University of Alberta（阿尔伯塔大学）
Modern surgeries are performed in complex and dynamic settings, including ever-changing interactions between medical staff, patients, and equipment. The holistic modeling of the operating room (OR) is, therefore, a challenging but essential task, with the potential to optimize the performance of surgical teams and aid in developing new surgical technologies to improve patient outcomes. The holistic representation of surgical scenes as semantic scene graphs (SGG), where entities are represented as nodes and relations between them as edges, is a promising direction for fine-grained semantic OR u…
Sterilizable Scene Graph Generation for Operating Rooms
Technical University of Darmstadt（达姆施塔特工业大学）；ImFusion GmbH（ImFusion有限公司）；Carl Zeiss AG（卡尔蔡司股份公司）；University Medical Center Mainz（美因茨大学医学中心）；Inria Center at University Côte d’Azur（蔚蓝海岸大学法国国家信息与自动化研究所中心）
Multi-view Video-Pose Pretraining for Operating Room Surgical Activity Recognition
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Intuitive Surgical Inc.（直觉外科公司）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms
The Chinese University of Hong Kong（香港中文大学）；Technical University of Denmark（丹麦技术大学）；Huazhong University of Science and Technology（华中科技大学）；National University of Singapore（新加坡国立大学）
Gaze-tracking is a novel way of interacting with computers which allows new scenarios, such as enabling people with motor-neuron disabilities to control their computers or doctors to interact with patient information without touching screen or keyboard. Further, there are emerging applications of gaze-tracking in interactive gaming, user experience research, human attention analysis and behavioral studies. Accurate estimation of the gaze may involve accounting for head-pose, head-position, eye rotation, distance from the object as well as operating conditions such as illumination, occlusion, b…
Mitigating Biases in Surgical Operating Rooms with Geometry
Computer Aided Medical Procedures, TU Munich, Germany（图宾根大学医学辅助技术部）；Munich Center for Machine Learning, Germany（慕尼黑机器学习中心）；UCL Hawkes Institute, Dept. Computer Science, University College London（伦敦大学学院霍克斯研究所）；Minimally Invasive Surgery, University Hospital of Munich (LMU), Germany（慕尼黑大学医院微创外科）
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University Hospital（海德堡大学医院）；University of Heidelberg（海德堡大学）；National Center for Tumor Diseases, Partner Site Dresden（国家肿瘤疾病中心德累斯顿分中心）；HIDSS4Health – Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；University of Calabria（卡拉布里亚大学）；understandAI GmbH（understandAI 有限公司）；University of Tübingen（蒂宾根大学）；BMW Group（宝马集团）；KARL STORZ SE & Co. KG（卡尔史托斯公司）；TU Dresden（德累斯顿工业大学）
Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a speci…
TeleOR: Real-time Telemedicine System for Full-Scene Operating Room
School of Public Health, Zhejiang University（浙江大学公共卫生学院）；The Second Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院附属第二医院）；Institute of Wenzhou, Zhejiang University（浙江大学温州研究院）；School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）理工学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Notre Dame（圣母大学）
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；IHU Strasbourg（斯特拉斯堡大学医院研究所）
The fine-grained localization of clinicians in the operating room (OR) is a key component to design the new generation of OR support systems. Computer vision models for person pixel-based segmentation and body-keypoints detection are needed to better understand the clinical activities and the spatial layout of the OR. This is challenging, not only because OR images are very different from traditional vision datasets, but also because data and annotations are hard to collect and generate in the OR due to privacy concerns. To address these concerns, we first study how joint person pose estimatio…
Beyond Role-Based Surgical Domain Modeling: Generalizable Re-Identification in the Operating Room
Chair for Computer Aided Medical Procedures, Technical University of Munich（计算机辅助医疗程序研究所，慕尼黑技术大学）；UCL Hawkes Institute, Dept. Computer Science, University College London（UCL哈维斯研究所，伦敦大学学院计算机科学系）；Department of General, Visceral, and Transplant Surgery, University Hospital, Ludwig-Maximilians-University（普通外科、 visceral外科及移植外科部门，路德维希-马克西米利安大学医院）
Self-Supervised Uncalibrated Multi-View Video Anonymization in the Operating Room
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学，法国国家科学研究中心，法国国家卫生研究院，ICube，UMR7357，法国）；University Hospital of Strasbourg, 67000 Strasbourg, France（斯特拉斯堡大学医院，法国）
SurgBox: Agent-Driven Operating Room Sandbox with Surgery Copilot
CAIR, HKISI, CAS（中国科学院香港中文大学信息安全研究所智能计算研究中心）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；Peking Union Medical College Hospital（北京协和医院）
Open, or non-laparoscopic surgery, represents the vast majority of all operating room procedures, but few tools exist to objectively evaluate these techniques at scale. Current efforts involve human expert-based visual assessment. We leverage advances in computer vision to introduce an automated approach to video analysis of surgical execution. A state-of-the-art convolutional neural network architecture for object detection was used to detect operating hands in open surgery videos. Automated assessment was expanded by combining model predictions with a fast object tracker to enable surgeon-sp…
Stony Brook University（石溪大学）；Temple University（天普大学）；South China University of Technology（华南理工大学）；Peng Cheng Laboratory（鹏城实验室）；HiScene Information Technologies（亮风台信息科技有限公司）；Amazon Web Services（亚马逊云科技）
Despite great recent advances in visual tracking, its further development, including both algorithm design and evaluation, is limited due to lack of dedicated large-scale benchmarks. To address this problem, we present LaSOT, a high-quality Large-scale Single Object Tracking benchmark. LaSOT contains a diverse selection of 85 object classes, and offers 1,550 totaling more than 3.87 million frames. Each video frame is carefully and manually annotated with a bounding box. This makes LaSOT, to our knowledge, the largest densely annotated tracking benchmark. Our goal in releasing LaSOT is to provi…
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学、法国国家科学研究中心、法国国家医学研究院、ICube、UMR7357、法国）；Department for Biomedical Research (DBMR), University of Bern, 3008 Bern, Switzerland（生物医学研究部（DBMR）、伯尔尼大学、瑞士伯尔尼3008、瑞士）；University Hospital of Strasbourg, 67000 Strasbourg, France（斯特拉斯堡大学医院、法国斯特拉斯堡67000、法国）；Department for Visceral Surgery（visceral surgery 部）；Medicine, Bern University Hospital, University of Bern, 3010 Bern, Switzerland（医学、伯尔尼大学医院、伯尔尼大学、瑞士伯尔尼3010、瑞士）
When do they StOP?: A First Step Towards Automatically Identifying Team Communication in the Operating Room
University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University of Bern（伯尔尼大学）；Clarunis（克拉鲁尼斯）；University of Basel（巴塞尔大学）；University Hospital of Strasbourg（斯特拉斯堡大学医院）；Bern University Hospital（伯尔尼大学医院）
Real-Time Brain Tumor Detection in Intraoperative Ultrasound Using YOLO11: From Model Training to Deployment in the Operating Room
Department of Neurosurgery, Rio Hortega University Hospital（里奥霍尔特加大学医院神经外科部）；Department of Neurosurgery, Tata Memorial Centre, Homi Bhabha National Institute（塔塔纪念中心神经外科部，霍米·巴布哈国家研究所）；Biomedical Engineering Group, University of Valladolid（瓦拉多利德大学生物医学工程小组）；UMR 1253, iBrain, Université de Tours, Inserm（UMR 1253，iBrain， Tours大学，Inserm）；Department of Neurosurgery, CHRU de Tours（图尔CHRU医院神经外科部）；Department of Neurosrugery, ARNAS Civico Di Cristina Benfratelli Hospital（ARNAS公民迪克里斯塔弗拉蒂医院神经外科部）；Fondazione IRCCS Istituto Neurologico Carlo Besta（IRCCS神经学研究所卡尔罗·贝斯塔基金会）；University of Milan（米兰大学）；Johns Hopkins Medical School（约翰霍普金斯医学院）
