arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.31245cs.CV

HyperVLP: 在双曲空间中增强层次化手术视频-语言预训练

HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space

Zhejiang University(浙江大学);University of Strasbourg, CNRS, INSERM, ICube, UMR7357(斯特拉斯堡大学,CNRS,INSERM,ICube,UMR7357);IHU Strasbourg(斯特拉斯堡IHU);Technical University of Munich(慕尼黑工业大学)

Yaojun Hu, Kun Yuan, Nassir Navab, Haochao Ying, Jian Wu, Nicolas Padoy

AI总结:

提出双曲空间手术视频-语言预训练框架HyperVLP,通过保留层次结构、缓解结构假负例并强制父子语义一致性,提升零样本和少样本手术阶段识别性能。

摘要:

手术视觉-语言基础模型通常采用教育材料(如手术讲座视频)将语言编码的手术知识转移到视觉表示中。这些知识是多维且层次化的:细粒度动作线索出现在旁白中,中级关键步骤总结在子节标题中,全局过程上下文(如患者病史和手术策略)在摘要文本中描述。先前的工作大多将这些异质信号压缩到一个单一的平坦嵌入空间中,隐含地假设层次级别之间的独立性。然而,这是次优的,因为它忽略了跨层级的语义包含关系(例如,动作属于步骤,步骤构成阶段),削弱了长距离依赖建模。为此,我们提出了一个双曲手术视频-语言预训练框架,通过缓解由过程上下文引起的结构假负例并强制父阶段与其组成子步骤之间的语义一致性,显式保留层次结构。在多个手术基准上的大量实验表明,在跨手术和机构的零样本和少样本阶段识别中取得了一致的性能提升。

英文摘要:

Surgical vision-language foundation models typically adopt educational materials, such as surgical lecture videos, to transfer surgical knowledge encoded in language into visual representations. These knowledge are multi-dimensional and hierarchical: fine-grained action cues appear in narration, mid-level key steps are summarized in subsection headings, and global procedural context, such as patient history and surgical strategy, is described in abstract texts. Prior work largely collapses these heterogeneous signals into a single flat embedding space, implicitly assuming independence across hierarchy levels. However, this is suboptimal because it ignores cross-level semantic containment, e.g., actions belong to steps, steps compose phases, weakens long-range dependency modeling. To this end, we propose a hyperbolic surgical video-language pre-training framework that explicitly preserves the hierarchical structure by mitigating structural false negatives induced by procedural context and enforcing semantic consistency between parent phases and their constituent child steps. Extensive experiments on multiple surgical benchmarks show consistent gains in zero- and few-shot phase recognition across procedures and institutions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑