SCOPE：用于手术场景分割的语音引导协同感知框架
SCOPE: Speech-guided COllaborative PErception Framework for Surgical Scene Segmentation
AI总结：
本文提出语音引导协同感知框架SCOPE，结合LLM推理与开放集VFM感知能力，利用临床医生语音反馈指导术中视频的手术器械与解剖结构实时分割、标注和跟踪，实现免提、以外科医生为中心的动态手术室人机AI协作。
摘要：
对手术场景相关元素进行准确的分割和跟踪，对于实现具备上下文感知能力的术中辅助和决策至关重要。当前的解决方案仍局限于特定领域的监督模型，这些模型依赖标注数据，且需要特定领域的数据才能适应新的手术场景以及超越预定义的标签类别。近期，提示驱动的视觉基础模型（VFM）的进展使得跨异构医学图像的开放集、零样本分割成为可能。然而，这些模型对手动视觉或文本提示的依赖限制了它们在术中环境中的部署。我们引入了一种语音引导的协同感知（SCOPE）框架，该框架将大型语言模型（LLM）的推理能力与开放集VFM的感知能力相集成，以支持对术中视频流中的手术器械和解剖结构进行实时分割、标注和跟踪。该框架的一个关键组件是协同感知智能体，它生成VFM分割的顶级候选结果，并结合临床医生直观的语音反馈，在自然的人机协作范式中指导手术器械的分割。随后，器械本身作为交互式指针来标注手术场景的其他元素。我们在公开可用的Cataract1k数据集子集和内部离体颅底数据集上评估了所提出的框架，以证明其生成手术场景实时分割和跟踪的潜力。此外，我们通过现场模拟离体实验展示了其动态能力。这种人机AI协作范式展示了为动态手术室环境开发可适应、免提、以外科医生为中心的工具的潜力。
英文摘要：
Accurate segmentation and tracking of relevant elements of the surgical scene is crucial to enable context-aware intraoperative assistance and decision making. Current solutions remain tethered to domain-specific, supervised models that rely on labeled data and required domain-specific data to adapt to new surgical scenarios and beyond predefined label categories. Recent advances in prompt-driven vision foundation models (VFM) have enabled open-set, zero-shot segmentation across heterogeneous medical images. However, dependence of these models on manual visual or textual cues restricts their deployment in introperative surgical settings. We introduce a speech-guided collaborative perception (SCOPE) framework that integrates reasoning capabilities of large language model (LLM) with perception capabilities of open-set VFMs to support on-the-fly segmentation, labeling and tracking of surgical instruments and anatomy in intraoperative video streams. A key component of this framework is a collaborative perception agent, which generates top candidates of VFM-generated segmentation and incorporates intuitive speech feedback from clinicians to guide the segmentation of surgical instruments in a natural human-machine collaboration paradigm. Afterwards, instruments themselves serve as interactive pointers to label additional elements of the surgical scene. We evaluated our proposed framework on a subset of publicly available Cataract1k dataset and an in-house ex-vivo skull-base dataset to demonstrate its potential to generate on-the-fly segmentation and tracking of surgical scene. Furthermore, we demonstrate its dynamic capabilities through a live mock ex-vivo experiment. This human-AI collaboration paradigm showcase the potential of developing adaptable, hands-free, surgeon-centric tools for dynamic operating-room environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
