SurgLLM：一种具备空间聚焦与时间感知的多用途手术视频理解大模型
SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding
AI总结：
针对现有手术视频理解模型空间感知不足、时间推理弱的问题，提出具备空间聚焦与时间感知的SurgLLM大模型，通过专用预训练、微调与任务动态集成，在多类手术视频理解任务上超越现有最优方法。
摘要：
手术视频理解对推动计算机辅助手术（Computer-Assisted Surgery，CAS）系统发展至关重要。尽管现有研究已取得显著进展，但仍存在两大核心局限：手术视频的视觉内容感知能力不足、时间感知能力欠缺，这阻碍了多用途CAS解决方案的开发。本研究提出SurgLLM框架，这是一款专为多类手术视频理解任务定制的高效大多模态模型，具备增强的空间聚焦与时间感知能力。具体而言，为强化手术视频的空间聚焦能力，我们首先为SurgLLM的视频编码器设计了手术上下文感知多模态预训练（Surgical Context-aware Multimodal Pretraining，Surg-Pretrain），通过执行以器械为中心的掩码视频重建（Masked Video Reconstruction，MV-Recon）及后续多模态对齐实现。为将手术时间知识融入SurgLLM，我们进一步提出时间感知多模态微调（Temporal-aware Multimodal Tuning，TM-Tuning），通过交错多模态嵌入增强时间推理能力。此外，为无冲突地适配各类手术视频理解任务，我们设计了手术任务动态集成模块，可在SurgLLM中利用最优可学习参数高效分流查询。我们在字幕生成、通用视觉问答（VQA）、时间视觉问答等多类手术视频理解任务上开展了大量实验，结果表明该方法相较现有最优（state-of-the-art）方案实现了显著性能提升，验证了SurgLLM在多用途手术视频理解中的有效性。源代码可访问https://github.com/franciszchen/SurgLLM获取。
英文摘要：
Surgical video understanding is crucial for facilitating Computer-Assisted Surgery (CAS) systems. Despite significant progress in existing studies, two major limitations persist, including inadequate visual content perception and insufficient temporal awareness in surgical videos, and hinder the development of versatile CAS solutions. In this work, we propose the SurgLLM framework, an effective large multimodal model tailored for versatile surgical video understanding tasks with enhanced spatial focus and temporal awareness. Specifically, to empower the spatial focus of surgical videos, we first devise Surgical Context-aware Multimodal Pretraining (Surg-Pretrain) for the video encoder of SurgLLM, by performing instrument-centric Masked Video Reconstruction (MV-Recon) and subsequent multimodal alignment. To incorporate surgical temporal knowledge into SurgLLM, we further propose Temporal-aware Multimodal Tuning (TM-Tuning) to enhance temporal reasoning with interleaved multimodal embeddings. Moreover, to accommodate various understanding tasks of surgical videos without conflicts, we devise a Surgical Task Dynamic Ensemble to efficiently triage a query with optimal learnable parameters in our SurgLLM. Extensive experiments performed on diverse surgical video understanding tasks, including captioning, general VQA, and temporal VQA, demonstrate significant improvements over the state-of-the-art approaches, validating the effectiveness of our SurgLLM in versatile surgical video understanding. The source code is available at https://github.com/franciszchen/SurgLLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
