arXiv 2506.178732026-02-03cs.CVcs.AI

SurgVidLM：迈向多粒度外科视频理解的大型语言模型

SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Model

The Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen University（中山大学）；University of Strasbourg（斯特拉斯堡大学）；Technical University of Munich（慕尼黑技术大学）；Monash University（墨尔本大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（人工智能与机器人中心，HKISI-CAS）；Shanghai AI Laboratory（上海人工智能实验室）

Guankun Wang, Junyi Wang, Wenjin Mo, Long Bai, Kun Yuan, Ming Hu, Jinlin Wu, Junjun He, Yiming Huang, Nicolas Padoy, Zhen Lei, Hongbin Liu, Nassir Navab, Hongliang Ren

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

SurgVidLM通过多粒度分析提升外科视频理解能力，结合全局与局部机制实现更精确的手术流程解析。

摘要：

外科场景理解对于外科培训和机器人决策在机器人辅助手术中至关重要。最近多模态大语言模型（MLLMs）的进展展示了在医学领域推进场景感知的巨大潜力，使外科医生能够理解和分析手术场景和流程。然而，这些方法主要面向基于图像的分析或全局视频理解，忽视了对特定过程进行细致分析和捕捉详细任务执行至关重要的细粒度视频推理。为弥补这一差距，我们提出了SurgVidLM，这是首个针对全粒度和细粒度外科视频理解设计的视频语言模型。为了训练我们的SurgVidLM，我们构建了SVU-31K，这是一个包含超过31,000个视频-指令对的大型数据集，能够实现手术流程的整体理解和详细分析。基于这一资源，SurgVidLM集成了一个两阶段的StageFocus机制：第一阶段提取全局流程上下文，而第二阶段则通过时间线索进行高频局部分析。我们还开发了多频率融合注意力机制，以有效整合低频和高频的视觉标记，确保关键任务特定细节的保留。实验结果表明，SurgVidLM在全粒度和细粒度视频理解任务中显著优于同等参数规模的最新Vid-LLMs，展示了其在捕捉复杂机器人辅助手术上下文方面的优越能力。我们的代码和数据集将很快公开。

英文摘要：

Surgical scene understanding is critical for surgical training and robotic decision-making in robot-assisted surgery. Recent advances in Multimodal Large Language Models (MLLMs) have demonstrated great potential for advancing scene perception in the medical domain, facilitating surgeons to understand surgical scenes and procedures. However, these methods are primarily oriented towards image-based analysis or global video understanding, overlooking the fine-grained video reasoning that is crucial for analyzing specific processes and capturing detailed task execution within a surgical procedure. To bridge this gap, we propose SurgVidLM, the first video language model designed to address both full and fine-grained surgical video comprehension. To train our SurgVidLM, we construct the SVU-31K that is a large-scale dataset with over 31K video-instruction pairs, enabling both holistic understanding and detailed analysis of surgical procedures. Building on this resource, SurgVidLM incorporates a two-stage StageFocus mechanism: the first stage extracts global procedural context, while the second stage performs high-frequency local analysis guided by temporal cues. We also develop the Multi-frequency Fusion Attention to effectively integrate low- and high-frequency visual tokens, ensuring the preservation of critical task-specific details. Experimental results demonstrate that SurgVidLM significantly outperforms state-of-the-art Vid-LLMs of comparable parameter scale in both full and fine-grained video understanding tasks, showcasing its superior capability in capturing the context of complex robot-assisted surgeries. Our code and dataset will be publicly accessible soon.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。