arXiv 2501.113472025-03-15cs.CV

EndoChat：用于内镜手术的扎根多模态大语言模型

EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery

The Chinese University of Hong Kong（香港中文大学）；Theory Lab, Central Research Institute, 2012 Labs, Huawei Technologies Co. Ltd.（华为技术有限公司中央研究院理论实验室2012实验室）；Chair of Computer Aided Procedures (CAMP), Technical University of Munich（慕尼黑工业大学计算机辅助程序教席）；Department of Gastroenterology, Qilu Hospital of Shandong University（山东大学齐鲁医院消化内科）

Guankun Wang, Long Bai, Junyi Wang, Kun Yuan, Zhen Li, Tianxu Jiang, Xiting He, Jinlin Wu, Zhen Chen, Zhen Lei, Hongbin Liu, Jiazheng Wang, Fan Zhang, Nicolas P… 展开作者

Guankun Wang, Long Bai, Junyi Wang, Kun Yuan, Zhen Li, Tianxu Jiang, Xiting He, Jinlin Wu, Zhen Chen, Zhen Lei, Hongbin Liu, Jiazheng Wang, Fan Zhang, Nicolas Padoy, Nassir Navab, Hongliang Ren

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对手术场景理解缺乏专用多模态大模型的问题，本文提出EndoChat，构建Surg-396K数据集并引入多尺度视觉交互与对比推理机制，在多项任务上达到SOTA并获得外科医生认可。

摘要：

近年来，多模态大语言模型（MLLMs）在计算机辅助诊断和决策中展现出巨大潜力。在机器人辅助手术场景下，MLLMs可作为手术培训与指导的有效工具。然而，目前仍缺乏专门面向临床应用中手术场景理解的MLLMs。本研究提出EndoChat，以应对外科医生在手术场景理解中遇到的多种对话范式和子任务。为训练EndoChat，我们通过一种新型流程构建了Surg-396K数据集；该流程基于收集的大规模内镜手术数据集，系统地提取手术信息并生成结构化标注。此外，我们提出多尺度视觉token交互机制和基于视觉对比的推理机制，以增强模型的表征学习与推理能力。我们的模型在五种对话范式和八项手术场景理解任务上均达到最先进性能。另外，我们还邀请专业外科医生进行评估，其中大多数人对与EndoChat协作给予了积极反馈。总体而言，这些结果表明EndoChat具有显著推动机器人辅助手术培训与自动化的巨大潜力。

英文摘要：

Recently, Multimodal Large Language Models (MLLMs) have demonstrated their immense potential in computer-aided diagnosis and decision-making. In the context of robotic-assisted surgery, MLLMs can serve as effective tools for surgical training and guidance. However, there is still a lack of MLLMs specialized for surgical scene understanding in clinical applications. In this work, we introduce EndoChat to address various dialogue paradigms and subtasks in surgical scene understanding that surgeons encounter. To train our EndoChat, we construct the Surg-396K dataset through a novel pipeline that systematically extracts surgical information and generates structured annotations based on collected large-scale endoscopic surgery datasets. Furthermore, we introduce a multi-scale visual token interaction mechanism and a visual contrast-based reasoning mechanism to enhance the model's representation learning and reasoning capabilities. Our model achieves state-of-the-art performance across five dialogue paradigms and eight surgical scene understanding tasks. Additionally, we conduct evaluations with professional surgeons, most of whom provide positive feedback on collaborating with EndoChat. Overall, these results demonstrate that our EndoChat has great potential to significantly advance training and automation in robotic-assisted surgery.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。