VS-Assistant：满足外科医生需求的多功能手术助手
VS-Assistant: Versatile Surgery Assistant on the Demand of Surgeons
AI总结：
针对现有手术辅助算法功能单一、需手动组合的问题，本研究提出基于多模态大语言模型的VS-Assistant，通过混合投影器模块和手术函数调用微调策略，可按需完成多类手术理解任务，在神经外科数据上性能优于现有模型。
摘要：
外科手术干预对患者医疗至关重要，已有诸多研究开发了先进算法，为外科医生提供手术理解与决策辅助。尽管取得了显著进展，但这些算法均针对单一特定任务与场景开发，实际应用中需要手动组合不同功能，限制了适用性。因此，业界亟需一款智能多功能手术助手，能够准确理解外科医生的意图，并据此执行特定任务以支撑手术流程。本研究借助先进的多模态大语言模型（MLLMs），提出了多功能手术助手VS-Assistant，可准确理解外科医生意图，按需完成一系列手术理解任务，例如手术场景分析、手术器械检测与分割。具体而言，为实现优异的手术多模态理解能力，我们设计了混合投影器（MOP）模块，用于对齐VS-Assistant中的手术多模态大语言模型，以平衡自然知识与手术知识。此外，我们设计了手术Function-Calling Tuning（函数调用微调）策略，使VS-Assistant能够理解手术意图，从而按需发起一系列手术功能调用，满足外科医生的需求。在神经外科数据上开展的大量实验证实，我们的VS-Assistant相比现有多模态大语言模型能更准确地理解外科医生意图，在文本分析与视觉任务中均取得了压倒性的性能优势。源代码与模型将对外公开。
英文摘要：
The surgical intervention is crucial to patient healthcare, and many studies have developed advanced algorithms to provide understanding and decision-making assistance for surgeons. Despite great progress, these algorithms are developed for a single specific task and scenario, and in practice require the manual combination of different functions, thus limiting the applicability. Thus, an intelligent and versatile surgical assistant is expected to accurately understand the surgeon's intentions and accordingly conduct the specific tasks to support the surgical process. In this work, by leveraging advanced multimodal large language models (MLLMs), we propose a Versatile Surgery Assistant (VS-Assistant) that can accurately understand the surgeon's intention and complete a series of surgical understanding tasks, e.g., surgical scene analysis, surgical instrument detection, and segmentation on demand. Specifically, to achieve superior surgical multimodal understanding, we devise a mixture of projectors (MOP) module to align the surgical MLLM in VS-Assistant to balance the natural and surgical knowledge. Moreover, we devise a surgical Function-Calling Tuning strategy to enable the VS-Assistant to understand surgical intentions, and thus make a series of surgical function calls on demand to meet the needs of the surgeons. Extensive experiments on neurosurgery data confirm that our VS-Assistant can understand the surgeon's intention more accurately than the existing MLLM, resulting in overwhelming performance in textual analysis and visual tasks. Source code and models will be made public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
