Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Northwestern University（西北大学）；University of Alberta（阿尔伯塔大学）；Yale University（耶鲁大学）；Nanfang Hospital（南华医院）；Shenzhen University of Advanced Technology（深圳大学先进技术研究院）
Jincai Huang, Shihao Zou, Yuchen Guo, Jingjing Li, Wei Ji, Kai Wang, Shanshan Wang, Weixin Si
Surgical scene understanding is a cornerstone of computer-assisted intervention. While recent advances, particularly in surgical image segmentation, have driven progress, real-world clinical applications require a more holistic understanding that jointly captures procedural context, semantic reasoning, and precise visual grounding. However, existing approaches typically address these components in isolation, leading to fragmented representations and limited semantic consistency. To address this limitation, we propose SurgMLLM, a unified surgical scene understanding framework that bridges high-level reasoning and low-level visual grounding within a single model. Given surgical videos, SurgMLLM fine-tunes a multimodal large language model (MLLM) to support structured interpretability reasoning, which is used to jointly model phases, instrument-verb-target (IVT) triplets, and triplet-entity segmentation tokens. These tokens are then temporally aggregated and serve as prompts for a segmentation network, enabling accurate pixel-wise grounding of triplet instruments and targets. The entire framework is trained end-to-end with a unified objective that couples language-based reasoning supervision with visual grounding losses, promoting coherent cross-task learning and clinically consistent scene representations. To facilitate unified evaluation, we introduce CholecT45-Scene, extending CholecT45 dataset with 64,299 frames of pixel-level mask annotations for instruments and targets, aligned with existing triplet labels. Extensive experiments show that SurgMLLM significantly advances surgical scene understanding, improving the primary triplet recognition metric AP_IVT from 40.7% to 46.0% and consistently outperforming prior methods in phase recognition and segmentation. These results highlight the effectiveness of unified reasoning-and-grounding for reliable, context-aware surgical assistance.
Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Department of Thoracic Surgery, the Seventh Affiliated Hospital, Sun Yat-sen University（中山大学第七附属医院胸外科部门）；Bioengineering/Imperial-X, Imperial College London（生物工程/Imperial-X，帝国理工学院伦敦分校）；ROAS Thrust, Hong Kong University of Science and Technology (Guangzhou)（ROAS项目，香港科技大学（广州））；Department of Electronic and Computer Engineering, Hong Kong SAR（香港特别行政区电子与计算机工程系）
SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark
School of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；University of Nottingham（诺丁汉大学）；School of AI, Shenzhen University（深圳大学人工智能学院）；Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University（广东省智能信息处理重点实验室）
VS-Assistant: Versatile Surgery Assistant on the Demand of Surgeons
Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS（香港科学院人工智能与机器人中心（CAIR））；The Chinese University of Hong Kong（香港中文大学）；King’s College London（伦敦国王学院）
MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；East China Normal University（东华大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Institute of High Performance Computing, A*STAR（高性能计算研究所，A*STAR）
HalluShift++: Bridging Language and Vision through Internal Representation Shifts for Hierarchical Hallucinations in MLLMs
Netaji Subhash Engineering College (NSEC)（奈尔贾伊·萨布哈工程学院）；TCG Crest；Electronics and Communication Sciences Unit (ECSU)（电子与通信科学单位）；Indian Statistical Institute（印度统计研究所）
EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery
The Chinese University of Hong Kong（香港中文大学）；Theory Lab, Central Research Institute, 2012 Labs, Huawei Technologies Co. Ltd.（华为技术有限公司中央研究院理论实验室2012实验室）；Chair of Computer Aided Procedures (CAMP), Technical University of Munich（慕尼黑工业大学计算机辅助程序教席）；Department of Gastroenterology, Qilu Hospital of Shandong University（山东大学齐鲁医院消化内科）
SurgVisAgent: Multimodal Agentic Model for Versatile Surgical Visual Enhancement
University of Chinese Academy of Sciences Multimedia Department, Xiaomi Inc Chinese Academy of Sciences, Institute of Automation Hong Kong Institute of Science \& Innovation, Chinese Academy of Sciences
NeuroABench: A Multimodal Evaluation Benchmark for Neurosurgical Anatomy Identification
Hong Kong Institute of Science and Innovation（香港科学与创新研究院）；The University of Hong Kong-Shenzhen Hospital（香港大学深圳医院）；Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）
SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Model
The Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen University（中山大学）；University of Strasbourg（斯特拉斯堡大学）；Technical University of Munich（慕尼黑技术大学）；Monash University（墨尔本大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（人工智能与机器人中心，HKISI-CAS）；Shanghai AI Laboratory（上海人工智能实验室）
PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；Pengcheng Laboratory（鹏城实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Data Science and Artificial Intelligence Research Institute, China United Network Communications Group Co., Ltd.（中国联合网络通信集团有限公司数据科学与人工智能研究院）
Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Department of Ophthalmology, The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院眼科学系）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）
EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgery
School of Computer Science & Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computer Science, University of Nottingham Ningbo China（诺丁汉大学宁波校区计算机科学学院）；Wenzhou Medical University（温州医科大学）；School of Computer Science, University of Nottingham（诺丁汉大学计算机科学学院）
Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench
School of Biomedical Engineering, Southern Medical University（生物医学工程学院，南方医科大学）；School of Biomedical Engineering, Shanghai Jiaotong University（生物医学工程学院，上海交通大学）；Department of Electronic Engineering, Chinese University of Hong Kong（电子工程系，中国香港大学）；Faculty of Dentistry, The University of Hong Kong（牙科学院，香港大学）；Department of Nuclear Medicine, The Second Affiliated Hospital of Guangzhou University of Chinese Medicine（核医学科，广州中医药大学第二附属医院）；PET Center, Department of Nuclear Medicine, Guangdong Provincial People’s Hospital, Southern Medical University（PET中心，核医学科，广东省人民医院，南方医科大学）；Department of Nuclear Medicine, Nanfang Hospital, Southern Medical University（核医学科，南芳医院，南方医科大学）；Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospitals（核医学与分子影像学部，日内瓦大学医院）；Departments of Radiology, Physics, and Biomedical Engineering, The University of British Columbia（放射学、物理和生物医学工程系，不列颠哥伦比亚大学）；Medical Artificial Intelligence Laboratory, Westlake University（医学人工智能实验室，西湖大学）
