手术视频中稀疏功能标志点定位的密集结构先验
Dense Structural Priors for Sparse Functional Landmark Localization in Surgical Videos
AI总结：
提出利用SAM 3作为结构先验，通过零样本点提示掩码和轻量级精炼框架，在无像素级标注下实现手术视频中功能标志点定位，F1达72.4%（尖端）和58.0%（锚点）。
摘要：
视觉基础模型如SAM 3能够在多样化的手术视频条件下提供可迁移的对象级结构，但分割输出并未明确编码定义功能性手术标志点的动作条件语义。估计器械范围和几何形状不同于定位与夹持、抓取或解剖相关的尖端或锚点。我们研究了视觉基础模型启用的稀疏动作感知标志点定位，利用零样本、点提示的结构掩码提供密集的器械级上下文，无需手动像素级掩码标注。我们提出一个轻量级精炼框架，使用SAM 3作为结构先验。一个粗粒度多帧网络预测尖端和锚点提示，生成非理想掩码，这些掩码与视觉和热图特征融合以精炼功能标志点预测。我们比较了直接掩码增强监督、预测导出掩码先验精炼和辅助掩码监督，以研究视觉基础模型导出的结构应如何进入一个精度导向的定位系统。在来自60个手术视频的7,867个剪辑上进行的实验，涵盖YouTube、Cholec80、HeiChole、SurgVU和CRCD，评估了该方法在异质条件下的表现。在无需手动像素级掩码标注进行训练的情况下，所提模型在尖端定位上达到72.4%的整体F1分数，在锚点定位上达到58.0%。直接将掩码施加于热图目标会使学习偏向于宽泛的工具区域，而预测导出先验和辅助监督则为动作依赖的标志点预测提供了有效的中间结构指导。
英文摘要：
Vision foundation models such as SAM 3 can provide transferable object-level structure across diverse surgical video conditions, but segmentation outputs do not explicitly encode the action-conditioned semantics that define functional surgical landmarks. Estimating instrument extent and geometry differs from localizing the tip or anchor relevant to clipping, grasping, or dissecting. We investigate vision foundation model-enabled sparse action-aware landmark localization, using zero-shot, point-prompted structural masks to provide dense instrument-level context without manual pixel-level mask annotations. We propose a lightweight refinement framework that uses SAM 3 as a structural prior. A coarse multi-frame network predicts tip and anchor prompts, generating non-oracle masks that are fused with visual and heatmap features to refine functional landmark predictions. We compare direct mask-augmented supervision, prediction-derived mask-prior refinement, and auxiliary mask supervision to examine how vision foundation model-derived structure should enter a precision-oriented localization system. Experiments on 7,867 clips from 60 surgical videos spanning YouTube, Cholec80, HeiChole, SurgVU, and CRCD evaluate the approach under heterogeneous conditions. Without manual pixel-level mask annotations for training, the proposed model achieves overall F1 scores of 72.4% for tip and 58.0% for anchor localization. Ablations show that coarse-to-fine refinement provides a substantial performance gain, while prediction-derived structural priors provide additional improvement when incorporated as intermediate guidance rather than direct localization targets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
