我们不断整合自身关于世界的知识与理解，以增强对所见内容的解读。这种能力在需要对多个实体和概念进行推理的应用领域至关重要，例如 AI 增强手术。本文提出了一种通过时间概念图网络将概念性知识整合到时间分析任务中的新方法。在所提出的网络中，全局知识图被纳入手术实例的时间分析，学习概念和关系在应用于数据时的含义。我们在手术视频数据上展示了结果，任务包括关键安全视野验证以及 Parkland 分级量表评估。结果表明，我们的方法改善了对复杂基准的识别和检测，并支持其他受关注的分析应用。
英文摘要：
We constantly integrate our knowledge and understanding of the world to enhance our interpretation of what we see.
This ability is crucial in application domains which entail reasoning about multiple entities and concepts, such as AI-augmented surgery. In this paper, we propose a novel way of integrating conceptual knowledge into temporal analysis tasks via temporal concept graph networks. In the proposed networks, a global knowledge graph is incorporated into the temporal analysis of surgical instances, learning the meaning of concepts and relations as they apply to the data. We demonstrate our results in surgical video data for tasks such as verification of critical view of safety, as well as estimation of Parkland grading scale. The results show that our method improves the recognition and detection of complex benchmarks as well as enables other analytic applications of interest.
SurgPETL: Parameter-Efficient Image-to-Surgical-Video Transfer Learning for Surgical Phase Recognition
Hong Kong University of Science and Technology（香港科技大学）；Tongji University（同济大学）；Tongji Hospital（同济医院）；Tongji Institute of Digestive Disease（同济消化病研究所）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
Token Merging via Spatiotemporal Information Mining for Surgical Video Understanding
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）；School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通信学院，华中科技大学）
The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology Shenzhen Research Institute（香港科技大学深圳研究院）
Self-supervised learning has witnessed great progress in vision and NLP; recently, it also attracted much attention to various medical imaging modalities such as X-ray, CT, and MRI. Existing methods mostly focus on building new pretext self-supervision tasks such as reconstruction, orientation, and masking identification according to the properties of medical images. However, the publicly available self-supervision models are not fully exploited. In this paper, we present a powerful yet efficient self-supervision framework for surgical video understanding. Our key insight is to distill knowled…
SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB)（电子、信息与生物工程系）；Politecnico di Milano（米兰理工大学）；IRCCS Humanitas Research Hospital（IRCCS人类itas研究医院）；UCL Hawkes Institute and Department of Computer Science（UCL Hawkes研究所和计算机科学系）；University College London（伦敦大学学院）；University of Manchester（曼彻斯特大学）
SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding
University of Liverpool（利物浦大学）；City University of Hong Kong（香港城市大学）；University of Oxford（牛津大学）；University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Imperial College London（帝国理工学院）
TU Munich（慕尼黑工业大学）；University Hospital of Munich (LMU)（慕尼黑大学医院（路德维希-马克西米利安-慕尼黑大学））；John Hopkins University（约翰斯·霍普金斯大学）
The surgical operating room (OR) presents many opportunities for automation and optimization. Videos from various sources in the OR are becoming increasingly available. The medical community seeks to leverage this wealth of data to develop automated methods to advance interventional care, lower costs, and improve overall patient outcomes. Existing datasets from OR room cameras are thus far limited in size or modalities acquired, leaving it unclear which sensor modalities are best suited for tasks such as recognizing surgical action from videos. This study demonstrates that surgical action reco…
Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning
University of Strasbourg（斯特拉斯堡大学）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡图像引导外科研究所（IHU-Strasbourg））；Fondazione Policlinico Universitario Agostino Gemelli IRCCS（阿戈斯蒂诺·杰梅利大学综合医院基金会IRCCS）；Institute for Research Against Digestive Cancer (IRCAD)（消化道癌症研究所（IRCAD））
University of Zagreb（萨格勒布大学）；University of Twente（特文特大学）；University of Groningen（格罗宁根大学）；Radboud University Medical Center（拉德堡德大学医学中心）；Canisius-Wilhelmina Hospital（卡尼修斯-威廉明娜医院）
Automatic recognition of surgical activities in the operating room (OR) is a key technology for creating next generation intelligent surgical devices and workflow monitoring/support systems. Such systems can potentially enhance efficiency in the OR, resulting in lower costs and improved care delivery to the patients. In this paper, we investigate automatic surgical activity recognition in robot-assisted operations. We collect the first large-scale dataset including 400 full-length multi-perspective videos from a variety of robotic surgery cases captured using Time-of-Flight cameras. We densely…
CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学，法国国家科学研究中心，法国国家医学研究院，ICube，UMR7357，法国）；IHU Strasbourg, France（斯特拉斯堡IHU，法国）；Technical University of Munich, Germany（慕尼黑技术大学，德国）
SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding
Hong Kong Institute of Science & Innovation（香港科学与工业创新研究院）；CAMP, Technische Universität München（CAMP，慕尼黑技术大学）；Department of Surgery, Faculty of Medicine, The Chinese University of Hong Kong（香港中文大学医学院外科部）
On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysis
National Center for Tumor Diseases (NCT/UCC)（国家肿瘤疾病中心（NCT/UCC））；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯医学院及大学医院）；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗森多夫亥姆霍兹中心）；Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology（德累斯顿工业大学人在回路触觉互联网中心）
SurgFed: Language-guided Multi-Task Federated Learning for Surgical Video Understanding
National University of Singapore（国立新加坡大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）
SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Model
The Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen University（中山大学）；University of Strasbourg（斯特拉斯堡大学）；Technical University of Munich（慕尼黑技术大学）；Monash University（墨尔本大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（人工智能与机器人中心，HKISI-CAS）；Shanghai AI Laboratory（上海人工智能实验室）
Modeling and automatically recognizing surgical activities are fundamental steps toward automation in surgery and play important roles in providing timely feedback to surgeons. Accurately recognizing surgical activities in video poses a challenging problem that requires an effective means of learning both spatial and temporal dynamics. Human gaze and visual saliency carry important information about visual attention and can be used to extract more relevant features that better reflect these spatial and temporal dynamics. In this study, we propose to use human gaze with a spatio-temporal attent…
StepAL: Step-aware Active Learning for Cataract Surgical Videos
Johns Hopkins University（约翰霍普金斯大学）；Wilmer Eye Institute（威尔默眼科研究所）；Johns Hopkins University School of Medicine（约翰霍普金斯大学医学院）；Malone Center for Engineering in Healthcare（马洛中心医疗工程中心）
HieraSurg: Hierarchy-Aware Diffusion Model for Surgical Video Generation
Chair for Computer Aided Medical Procedures (CAMP), TU Munich, Germany（计算机辅助医疗程序研究所（CAMP），慕尼黑技术大学，德国）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
Purpose: Basic surgical skills of suturing and knot tying are an essential part of medical training. Having an automated system for surgical skills assessment could help save experts time and improve training efficiency. There have been some recent attempts at automated surgical skills assessment using either video analysis or acceleration data. In this paper, we present a novel approach for automated assessment of OSATS based surgical skills and provide an analysis of different features on multi-modal data (video and accelerometer data). Methods: We conduct the largest study, to the best of o…
OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding
Monash University（莫纳什大学）；Airdoc-Monash Research（Airdoc-莫纳什研究所）；Airdoc；Bosch Corporate Research（博世企业研究院）；Cornell University（康奈尔大学）；King’s College London（伦敦国王学院）；Institute for Infocomm Research, A*STAR（新加坡资讯通信研究院（新加坡科技研究局下属））；City University of Macau（澳门城市大学）；Eye Hospital, Wenzhou Medical University（温州医科大学附属眼视光医院）
Activity recognition in surgical videos is a key research area for developing next-generation devices and workflow monitoring systems. Since surgeries are long processes with highly-variable lengths, deep learning models used for surgical videos often consist of a two-stage setup using a backbone and temporal sequence model. In this paper, we investigate many state-of-the-art backbones and temporal models to find architectures that yield the strongest performance for surgical activity recognition. We first benchmark the models performance on a large-scale activity recognition dataset containin…
NCT/UCC Dresden（德累斯顿国家肿瘤疾病中心/大学癌症中心）；CeTI, TU Dresden（德累斯顿工业大学触觉互联网卓越中心）；University Hospital Dresden（德累斯顿大学医院）
Research in unpaired video translation has mainly focused on short-term temporal consistency by conditioning on neighboring frames. However for transfer from simulated to photorealistic sequences, available information on the underlying geometry offers potential for achieving global consistency across views. We propose a novel approach which combines unpaired image translation with neural rendering to transfer simulated to photorealistic surgical abdominal scenes. By introducing global learnable textures and a lighting-invariant view-consistency loss, our method produces consistent translation…
Five billion people in the world lack access to quality surgical care. Surgeon skill varies dramatically, and many surgical patients suffer complications and avoidable harm. Improving surgical training and feedback would help to reduce the rate of complications, half of which have been shown to be preventable. To do this, it is essential to assess operative skill, a process that currently requires experts and is manual, time consuming, and subjective. In this work, we introduce an approach to automatically assess surgeon performance by tracking and analyzing tool movements in surgical videos, …
HTT-Net: Hierarchical Text-guided Transition Modeling for Surgical Video Phase Recognition
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Department of General Surgery, Beijing Friendship Hospital, Capital Medical University（首都医科大学附属北京友谊医院普通外科）；Chinese PLA General Hospital（中国人民解放军总医院）
SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery
University of Michigan（密歇根大学）；University of Edinburgh（爱丁堡大学）；Vanderbilt University Medical Center（范德比尔特大学医学中心）；University of Colorado（科罗拉多大学）；University of Rochester（罗切斯特大学）；Michigan Medicine（密歇根医学）
Hierarchical Curriculum Learning for Surgical Video Generation: Focusing on Bilateral Axillo-Breast Approach Robotic Thyroidectomy Datasets
Seoul National University College of Medicine（首尔国立大学医学院）；Seoul National University Hospital（首尔国立大学医院）；YONSEI WITH-YOU Surgery Clinic（延世WITH-YOU外科诊所）；Institute of Convergence Medicine with Innovative Technology, Seoul National University Hospital（首尔国立大学医院创新技术融合医学研究所）
Systematic Evaluation and Guidelines for Segment Anything Model in Surgical Video Analysis
Global College（全球学院）；Shanghai Jiao Tong University（上海交通大学）；Department of Automation（自动化系）；National University of Singapore（新加坡国立大学）；Department of Computer Science and Engineering（计算机科学与工程系）
Hi-GaTA: Hierarchical Gated Temporal Aggregation Adapter for Surgical Video Report Generation
School of Engineering, College of Engineering and Physical Sciences, University of Birmingham, Birmingham, UK（英国伯明翰大学工程学院）；School of Computer Science, University of Birmingham, Birmingham, UK（英国伯明翰大学计算机科学学院）；Department of Applied Health Sciences, University of Birmingham, Birmingham, UK（英国伯明翰大学应用健康科学系）
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）
Purpose: The objective of this investigation is to provide a comprehensive analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room. Methods: Using a data set of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluate feature based methods previously developed for surgical skill assessment mostly under benchtop settings. In addition, we present and validate two deep learning methods that directly assess skill using RGB videos. In the first method, we predict instrument tips as keypoints, and learn surgical skill using temporal…
A spatio-temporal network for video semantic segmentation in surgical videos
Medtronic Digital Surgery（美敦力数字外科）；Wellcome/EPSRC Centre for Internventional and Surgical Sciences, University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科学中心）
HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space
Zhejiang University（浙江大学）；University of Strasbourg, CNRS, INSERM, ICube, UMR7357（斯特拉斯堡大学，CNRS，INSERM，ICube，UMR7357）；IHU Strasbourg（斯特拉斯堡IHU）；Technical University of Munich（慕尼黑工业大学）
Scaling Video Pretraining for Surgical Foundation Models
The Johns Hopkins University（约翰霍普金斯大学）；Zhejiang University（浙江大学）；Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学伊利诺伊大学厄巴纳-香槟分校联合学院）；Zhejiang Lab（之江实验室）；Shaoxing University（绍兴大学）；Angelalign（时代天使）；Harvard Medical School（哈佛医学院）
OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining
Shanghai AI Laboratory（上海人工智能实验室）；University of Strasbourg（斯特拉斯堡大学）；Monash University（莫纳什大学）；University of Michigan（密歇根大学）；Shanghai Jiao Tong University（上海交通大学）；Xiamen University（厦门大学）；The Hong Kong Polytechnic University（香港理工大学）；Technische Universität München（慕尼黑工业大学）
SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos
Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China（人工智能与机器人中心，香港科学与创新研究院，中国科学院，香港，中国）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院，北京，中国）；Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany（医学辅助程序，慕尼黑技术大学，德国慕尼黑）；Electronic Engineering Department, The Chinese University of Hong Kong, Hong Kong, China（电子工程系，香港中文大学，香港，中国）；Neuromedical Centre, Hong Kong University Shenzhen Hospital, Shenzhen, China（神经医学中心，香港大学深圳医院，深圳，中国）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）；Department of Respiratory Medicine, The First Affiliated Hospital of Sun Yat-sen University, Guangzhou, China（呼吸科，中山大学附属第一医院，广州，中国）
National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden（德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯医学院暨大学医院）
Purpose: A profound education of novice surgeons is crucial to ensure that surgical interventions are effective and safe. One important aspect is the teaching of technical skills for minimally invasive or robot-assisted procedures. This includes the objective and preferably automatic assessment of surgical skill. Recent studies presented good results for automatic, objective skill evaluation by collecting and analyzing motion data such as trajectories of surgical instruments. However, obtaining the motion data generally requires additional equipment for instrument tracking or the availability …
