从相位接地到智能手术叙述
From Phase Grounding to Intelligent Surgical Narratives
AI总结：
本文提出基于CLIP的多模态框架，通过自动创建手术时间线和叙述，减少手动标注的需要。
摘要：
手术视频时间线是工具辅助手术的重要组成部分，因为它们允许外科医生快速聚焦于手术的关键部分。当前的方法涉及外科医生填写术后（OP）报告，这通常很模糊，或者手动标注手术视频，这非常耗时。我们提出的方法介于这两个极端之间：我们旨在从手术视频直接自动创建手术时间线和叙述。为此，我们采用基于CLIP的多模态框架，将手术视频帧与文本手势描述对齐。具体来说，我们使用CLIP视觉编码器从手术视频帧中提取表示，并使用文本编码器将相应的手势句子嵌入到共享嵌入空间中。然后我们对模型进行微调，以提高视频手势和文本标记之间的对齐程度。一旦训练完成，模型会预测视频帧的手势和阶段，从而构建结构化的手术时间线。这种方法利用预训练的多模态表示来连接视觉手势和文本叙述，减少外科医生手动视频审查和标注的需要。
英文摘要：
Video surgery timelines are an important part of tool-assisted surgeries, as they allow surgeons to quickly focus on key parts of the procedure. Current methods involve the surgeon filling out a post-operation (OP) report, which is often vague, or manually annotating the surgical videos, which is highly time-consuming. Our proposed method sits between these two extremes: we aim to automatically create a surgical timeline and narrative directly from the surgical video. To achieve this, we employ a CLIP-based multi-modal framework that aligns surgical video frames with textual gesture descriptions. Specifically, we use the CLIP visual encoder to extract representations from surgical video frames and the text encoder to embed the corresponding gesture sentences into a shared embedding space. We then fine-tune the model to improve the alignment between video gestures and textual tokens. Once trained, the model predicts gestures and phases for video frames, enabling the construction of a structured surgical timeline. This approach leverages pretrained multi-modal representations to bridge visual gestures and textual narratives, reducing the need for manual video review and annotation by surgeons.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
