Automated recognition of surgical phases and steps is a fundamental capability for intraoperative decision support, workflow automation, and skill assessment in robotic-assisted surgery. Existing approaches either depend on large-scale annotated surgical datasets or require expensive domain-specific pretraining on thousands of labeled videos, limiting their practical deployability across diverse robotic platforms and clinical environments. In this work, we propose TASOT (Text-Augmented Action Segmentation Optimal Transport), an annotation-free framework for surgical temporal segmentation that requires no task-specific annotations or surgical-domain pretraining. TASOT extends the Action Segmentation Optimal Transport (ASOT) formulation by incorporating temporally aligned textual descriptions generated directly from the input video, fusing visual and semantic cues within a unified unbalanced Gromov-Wasserstein optimal transport objective. Visual representations are extracted using DINOv3, while temporal captions produced by a vision-language model are encoded via CLIP and temporally aligned to individual frames, providing complementary semantic structure to the transport cost. We evaluate TASOT on three public surgical datasets and four benchmark settings spanning laparoscopic and robotic procedures, showing substantial improvements over the strongest zero-shot baselines: +18.9 F1 on Cholec80, +33.7 on AutoLaparo, +23.7 on StrasByPass70, and +4.5 on BernByPass70. These results suggest that fine-grained surgical workflow understanding in robotic settings can be achieved without manual training annotations or surgical-specific pretraining pipelines, offering a promising alternative for real-world robotic surgical systems.
SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding
Samsung Research（三星研究所）；Center for Humanoid Research, Korea Institute of Science and Technology（人类研究学院，韩国科学技术院）；Department of plastic surgery, College of medicine, Korea University（医学院整形外科部，韩国大学）；Department of orthopedic surgery, College of medicine, Korea University（医学院骨科部，韩国大学）
Surgical-MambaLLM: Mamba2-enhanced Multimodal Large Language Model for VQLA in Robotic Surgery
Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Imperial College London（帝国理工学院伦敦分校）；The Hong Kong University of Science and Technology（香港科学与技术大学）
SurgVisAgent: Multimodal Agentic Model for Versatile Surgical Visual Enhancement
University of Chinese Academy of Sciences Multimedia Department, Xiaomi Inc Chinese Academy of Sciences, Institute of Automation Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences
SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding
Hong Kong Institute of Science & Innovation（香港科学与工业创新研究院）；CAMP, Technische Universität München（CAMP，慕尼黑技术大学）；Department of Surgery, Faculty of Medicine, The Chinese University of Hong Kong（香港中文大学医学院外科部）
Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Department of Thoracic Surgery, the Seventh Affiliated Hospital, Sun Yat-sen University（中山大学第七附属医院胸外科部门）；Bioengineering/Imperial-X, Imperial College London（生物工程/Imperial-X，帝国理工学院伦敦分校）；ROAS Thrust, Hong Kong University of Science and Technology (Guangzhou)（ROAS项目，香港科技大学（广州））；Department of Electronic and Computer Engineering, Hong Kong SAR（香港特别行政区电子与计算机工程系）
Think Step by Step: Chain-of-Gesture Prompting for Error Detection in Robotic Surgical Videos
Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）；University College London（伦敦大学学院）；Wellcome/EPSRC Centre for Interventional and Surgical Sciences（Wellcome/EPSRC介入与外科科学中心）
Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery
The Chinese University of Hong Kong（香港中文大学）；Universiti Malaya（马来亚大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港智能系统研究所人工智能与机器人中心）；University College London（伦敦大学学院）
Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery
National University of Singapore（新加坡国立大学）；The Chinese University of Hong Kong (CUHK)（香港中文大学）；University College London（伦敦大学学院）；Shun Hing Institute of Advanced Engineering（信兴高等工程研究所）
Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）；University College London（伦敦大学学院）；National University of Singapore（新加坡国立大学）
Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery
Department of Biomedical Engineering, Eindhoven University of Technology（埃因霍温理工大学生物医学工程系）；Department of Electrical Engineering, Eindhoven University of Technology（埃因霍温理工大学电气工程系）；Department of Surgery, University Medical Center Utrecht（乌得勒支大学医学中心外科）
SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence
National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；University of Oxford（牛津大学）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）；Shanghai Jiao Tong University（上海交通大学）；Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）
EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Theory Lab, Central Research Institute, 2012 Labs, Huawei Technologies Co. Ltd.（华为技术有限公司中央研究院理论实验室2012实验室）；Chair of Computer Aided Procedures (CAMP), Technical University of Munich（慕尼黑工业大学计算机辅助程序教席）；Department of Gastroenterology, Qilu Hospital of Shandong University（山东大学齐鲁医院消化内科）
Voice-Interactive Surgical Agent for Multimodal Patient Data Control
Image Guided Precision Cancer Surgery Institute, College of Medicine, Korea University（影像引导精准癌症手术研究所，韩国大学医学院）；Department of Radiology, Massachusetts General Hospital（放射科，麻省总医院）；Department of Thoracic and Cardiovascular Surgery, Korea University Guro Hospital, College of Medicine, Korea University（胸外科和心血管外科，韩国大学Guro医院，韩国大学医学院）；Department of Biomedical Sciences, College of Medicine, Korea University（生物医学科学系，韩国大学医学院）
CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Qilu Hospital of Shandong University（山东大学齐鲁医院）；National University o（国立大学）
The Chinese University of Hong Kong（香港中文大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Reconstructing the scene of robotic surgery from the stereo endoscopic video is an important and promising topic in surgical data science, which potentially supports many applications such as surgical visual perception, robotic surgery education and intra-operative context awareness. However, current methods are mostly restricted to reconstructing static anatomy assuming no tissue deformation, tool occlusion and de-occlusion, and camera movement. However, these assumptions are not always satisfied in minimal invasive robotic surgeries. In this work, we present an efficient reconstruction pipel…
SurgicalGS: Dynamic 3D Gaussian Splatting for Accurate Robotic-Assisted Surgical Scene Reconstruction
The Hamlyn Centre for Robotic Surgery, Imperial College London（伦敦帝国理工学院哈姆林 robotic外科中心）；Hawkes Institute, University College London（伦敦大学学院霍克斯研究所）；University of Liverpool（利物浦大学）
Imperial College London（帝国理工学院）；German Aerospace Center（德国航空航天中心）；Technical University of Munich（慕尼黑工业大学）；Charing Cross Hospital（查令十字医院）
The aim of this paper is to introduce a robotic platform for autonomous iUS tissue scanning to optimise intraoperative diagnosis and improve surgical resection during robot-assisted operations. To guide anatomy specific robotic scanning and generate a representation of the robot task space, fast and accurate techniques for the recovery of 3D morphological structures of the surgical cavity are developed. The prototypic DLR MIRO surgical robotic arm is used to control the applied force and the in-plane motion of the US transducer. A key application of the proposed platform is the scanning of bra…
University of California San Diego（加州大学圣迭戈分校）；East China Normal University（华东师范大学）
Robotic automation in surgery requires precise tracking of surgical tools and mapping of deformable tissue. Previous works on surgical perception frameworks require significant effort in developing features for surgical tool and tissue tracking. In this work, we overcome the challenge by exploiting deep learning methods for surgical perception. We integrated deep neural networks, capable of efficient feature extraction, into the tissue reconstruction and instrument pose estimation processes. By leveraging transfer learning, the deep learning based approach requires minimal training data and re…
Robot-assisted surgery is an established clinical practice. The automatic identification of surgical actions is needed for a range of applications, including performance assessment of trainees and surgical process modeling for autonomous execution and monitoring. However, supervised action identification is not feasible, due to the burden of manually annotating recordings of potentially complex and long surgical executions. Moreover, often few example executions of a surgical procedure can be recorded. This paper proposes a novel fast algorithm for unsupervised identification of surgical actio…
Technische Universität München（慕尼黑工业大学）；Google（谷歌）；Johns Hopkins University（约翰斯·霍普金斯大学）
From a computer science viewpoint, a surgical domain model needs to be a conceptual one incorporating both behavior and data. It should therefore model actors, devices, tools, their complex interactions and data flow. To capture and model these, we take advantage of the latest computer vision methodologies for generating 3D scene graphs from camera views. We then introduce the Multimodal Semantic Scene Graph (MSSG) which aims at providing a unified symbolic, spatiotemporal and semantic representation of surgical procedures. This methodology aims at modeling the relationship between different c…
MIT（麻省理工学院）；Massachusetts General Hospital（麻省总医院）；Olympus Corporation（奥林巴斯公司）
Comprehension of surgical workflow is the foundation upon which artificial intelligence (AI) and machine learning (ML) holds the potential to assist intraoperative decision-making and risk mitigation. In this work, we move beyond mere identification of past surgical phases, into the prediction of future surgical steps and specification of the transitions between them. We use a novel Generative Adversarial Network (GAN) formulation to sample future surgical phases trajectories conditioned on past video frames from laparoscopic cholecystectomy (LC) videos and compare it to state-of-the-art appro…
Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University College London（伦敦大学学院）；The Hong Kong University of Science and Technology（香港科技大学）
University of California San Diego（加利福尼亚大学圣地亚哥分校）；Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information…
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University College London（伦敦大学学院）；Zhejiang Lab（之江实验室）
Surgical scene segmentation is fundamentally crucial for prompting cognitive assistance in robotic surgery. However, pixel-wise annotating surgical video in a frame-by-frame manner is expensive and time consuming. To greatly reduce the labeling burden, in this work, we study semi-supervised scene segmentation from robotic surgical video, which is practically essential yet rarely explored before. We consider a clinically suitable annotation situation under the equidistant sampling. We then propose PGV-CL, a novel pseudo-label guided cross-video contrast learning method to boost scene segmentati…
The Chinese University of Hong Kong（香港中文大学）；Shun Hing Institute of Advanced Engineering, CUHK（香港中文大学晨兴先进工程研究所）；T Stone Robotics Institute, CUHK（香港中文大学T Stone机器人研究所）
Performing low hertz labeling for surgical videos at intervals can greatly releases the burden of surgeons. In this paper, we study the semi-supervised instrument segmentation from robotic surgical videos with sparse annotations. Unlike most previous methods using unlabeled frames individually, we propose a dual motion based method to wisely learn motion flows for segmentation enhancement by leveraging temporal dynamics. We firstly design a flow predictor to derive the motion for jointly propagating the frame-label pairs given the current labeled frame. Considering the fast instrument motion, …
Hierarchical Curriculum Learning for Surgical Video Generation: Focusing on Bilateral Axillo-Breast Approach Robotic Thyroidectomy Datasets
Seoul National University College of Medicine（首尔国立大学医学院）；Seoul National University Hospital（首尔国立大学医院）；YONSEI WITH-YOU Surgery Clinic（延世WITH-YOU外科诊所）；Institute of Convergence Medicine with Innovative Technology, Seoul National University Hospital（首尔国立大学医院创新技术融合医学研究所）
Self-supervised, multi-modal learning has been successful in holistic representation of complex scenarios. This can be useful to consolidate information from multiple modalities which have multiple, versatile uses. Its application in surgical robotics can lead to simultaneously developing a generalised machine understanding of the surgical process and reduce the dependency on quality, expert annotations which are generally difficult to obtain. We develop a self-supervised, multi-modal representation learning paradigm that learns representations for surgical gestures from video and kinematics. …
University College London（伦敦大学学院）；Katholieke Universiteit Leuven（天主教鲁汶大学）；University College London Hospitals（伦敦大学学院医院）；Universitair Ziekenhuis Leuven（鲁汶大学医院）；Wellcome / EPSRC Centre for Surgical and Interventional Sciences, UCL, London, UK（惠康/英国工程与物理科学研究委员会外科与介入科学中心（伦敦大学学院））；Surgical Robot Vision Group, CMIC, Dept. Computer Science, University College London（伦敦大学学院计算机科学系CMIC外科机器人视觉小组）；Translational Imaging Group, CMIC, Dept. Med. Phys. Biomed. Eng., University College London（伦敦大学学院医学物理与生物医学工程系CMIC转化成像小组）
Real-time tool segmentation from endoscopic videos is an essential part of many computer-assisted robotic surgical systems and of critical importance in robotic surgical data science. We propose two novel deep learning architectures for automatic segmentation of non-rigid surgical instruments. Both methods take advantage of automated deep-learning-based multi-scale feature extraction while trying to maintain an accurate segmentation quality at all resolutions. The two proposed methods encode the multi-scale constraint inside the network architecture. The first proposed architecture enforces it…
Imperial College London（伦敦帝国理工学院）；Tsinghua University（清华大学）；NYU Abu Dhabi（纽约大学阿布扎比分校）；Wuhan University（武汉大学）；University College London（伦敦大学学院）；University of Liverpool（利物浦大学）
hutom；Yonsei University College of Medicine（延世大学医学院）
Automated surgical instrument localization is an important technology to understand the surgical process and in order to analyze them to provide meaningful guidance during surgery or surgical index after surgery to the surgeon. We introduce a new dataset that reflects the kinematic characteristics of surgical instruments for automated surgical instrument localization of surgical videos. The hSDB(hutom Surgery DataBase)-instrument dataset consists of instrument localization information from 24 cases of laparoscopic cholecystecomy and 24 cases of robotic gastrectomy. Localization information for…
Autonomy is the frontier of research in robotic surgery and its aim is to improve the quality of surgical procedures in the next future. One fundamental requirement for autonomy is advanced perception capability through vision sensors. In this paper, we propose a novel calibration technique for a surgical scenario with da Vinci robot. Calibration of the camera and the robot is necessary for precise positioning of the tools in order to emulate the high performance surgeons. Our calibration technique is tailored for RGB-D camera. Different tests performed on relevant use cases for surgery prove …
Surgical-DeSAM: Decoupling SAM for Instrument Segmentation in Robotic Surgery
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS)（Wellcome/EPSRC介入与外科科学中心（WEISS））；University College London（伦敦大学学院）；Dept. of Computer Science, University College London（伦敦大学学院计算机科学系）
National University of Singapore（新加坡国立大学）；Imperial College London（伦敦帝国学院）；Singapore General Hospital（新加坡中央医院）
Learning to infer graph representations and performing spatial reasoning in a complex surgical environment can play a vital role in surgical scene understanding in robotic surgery. For this purpose, we develop an approach to generate the scene graph and predict surgical interactions between instruments and surgical region of interest (ROI) during robot-assisted surgery. We design an attention link function and integrate with a graph parsing network to recognize the surgical interactions. To embed each node with corresponding neighbouring node features, we further incorporate SageConv into the …
The Chinese University of Hong Kong（香港中文大学）；T stone Robotics Institute（T石头机器人研究所）；Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems（粤港澳人机协同智能系统联合实验室）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Surgical instrument segmentation in robot-assisted surgery (RAS) - especially that using learning-based models - relies on the assumption that training and testing videos are sampled from the same domain. However, it is impractical and expensive to collect and annotate sufficient data from every new domain. To greatly increase the label efficiency, we explore a new problem, i.e., adaptive instrument segmentation, which is to effectively adapt one source model to new robotic surgical videos from multiple target domains, only given the annotated instruments in the first frame. We propose MDAL, a…
Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglement
Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）；Chair for Computer Aided Medical Procedures, Technical University of Munich（慕尼黑技术大学计算机辅助医疗程序主席职位）；Department of Biomedical Engineering, University of Toronto（多伦多大学生物医学工程系）；Center for Computational and Molecular Biology, Brown University（布朗大学计算与分子生物学中心）
Asan Medical Center（峨山医疗中心）；Johns Hopkins University（约翰斯·霍普金斯大学）；Chungnam National University School of Medicine（忠南大学医学院）
Some endoscopic surgeries require a surgeon to hold the endoscope with one hand and the surgical instruments with the other hand to perform the actual surgery with correct vision. Recent technical advances in deep learning as well as in robotics can introduce robotics to these endoscopic surgeries. This can have numerous advantages by freeing one hand of the surgeon, which will allow the surgeon to use both hands and to use more intricate and sophisticated techniques. Recently, deep learning with convolutional neural network achieves state-of-the-art results in computer vision. Therefore, the …
Center for Research and Formation in Artificial Intelligence, Universidad de los Andes（安第斯大学人工智能研究与培训中心）
We study the task of semantic segmentation of surgical instruments in robotic-assisted surgery scenes. We propose the Instance-based Surgical Instrument Segmentation Network (ISINet), a method that addresses this task from an instance-based segmentation perspective. Our method includes a temporal consistency module that takes into account the previously overlooked and inherent temporal information of the problem. We validate our approach on the existing benchmark for the task, the Endoscopic Vision 2017 Robotic Instrument Segmentation Dataset, and on the 2018 version of the dataset, whose anno…
