arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.16618cs.CVeess.IVq-bio.TO
收藏

通过语义长期跟踪实现手术视频的分割:SAM2S

SurgSLOT: Segment Anything in Surgical Videos via Semantic Long-term Tracking

National University of Singapore(新加坡国立大学);University of Sheffield(谢菲尔德大学)

Haofeng Liu, Ziyue Wang, Sudhanshu Mishra, Mingqi Gao, Guanyi Qin, Chang Han Low, Alex Y. W. Kong, Zhu Zhuo, Huazhu Fu, Joseph S. Ng, Yueming Jin

AI总结:

SAM2S通过语义长期跟踪提升手术视频分割性能,实现更准确的零样本泛化和实时推理

摘要:

手术视频分割对于计算机辅助手术至关重要,能够实现手术器械和组织的精确定位和跟踪。交互式视频对象分割(iVOS)模型如分割任何模型2(SAM2)提供了基于提示的灵活性,超越了具有预定义类别的方法,但在手术场景中面临挑战,由于领域差距和有限的长期跟踪能力。为了解决这些限制,我们构建了SA-SV,最大的手术iVOS基准,具有实例级时空注释(masklets),覆盖八种手术类型(61k帧,1.6k masklets),使长期跟踪和零样本泛化的发展和评估成为可能。基于SA-SV,我们提出了SAM2S,一个基础模型,通过:(1)DiveMem,一个可训练的多样化记忆机制,用于鲁棒的长期跟踪;(2)时间语义学习用于器械理解;以及(3)抗歧义学习以缓解多源数据集中的注释不一致。广泛的实验表明,通过SA-SV进行微调可带来显著的性能提升,SAM2在vanilla SAM2上平均提高了12.99平均J&F。SAM2S进一步将性能提升到80.42平均J&F,超越vanilla和微调SAM2分别17.10和4.11点,同时保持68 FPS的实时推理和强大的零样本泛化。代码和数据集将在https://jinlab-imvr.github.io/SAM2S上发布。

英文摘要:

Surgical scene understanding demands temporally consistent tracking of instruments and tissues. For clinical use, such tracking should generalize to new centers and procedure types, yet retraining for each of them is costly and not scalable. Interactive video object segmentation offers a way toward this generalization: the target is specified at inference by a first-frame visual prompt, so a model can generalize to unseen categories and new scenarios without retraining. However, training such a generalizable model demands spatio-temporal masklet annotations at a scale and procedural diversity that existing surgical benchmarks lack. We fill this gap with iSurg, the largest surgical segmentation benchmark to our knowledge, spanning six procedure types with over 170k frames, 410k object masks, and 2.4k masklets, including an in-house clinical dataset of four 30-minute videos. Yet tracking over such long videos remains challenging: the target needs to be re-identified among visually similar objects after long absences, while long-term memory has to judge whether each stored frame genuinely depicts the target, both demanding a stable object-level semantic identity. To this end, we propose SurgSLOT, a surgical segmentation generalist that segments any prompted target through two coupled modules built on this semantic identity: Temporal Semantic Learning learns it for re-identification, and Semantic-driven Long-term Memory reuses it to select reliable memory frames, suppressing identity drift over long procedures. On the SAM2 and SAM3 backbones, SurgSLOT reaches 81.0 and 82.8 Macro Average J&F under cross-dataset evaluation, surpassing their fine-tuned counterparts by 5.1 and 5.3 points and transferring zero-shot to an unseen procedure type and unseen object categories, with the SAM2 version running in real time at 68 FPS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑