通过多模态视觉-时间Transformer和生成式AI增强手术文档记录
Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI
AI总结：
提出一种结合视觉Transformer和大语言模型的多模态框架，通过提取帧级视觉特征、生成帧级描述并聚合时间特征，在CholecT50数据集上实现了高精度的手术视频自动摘要与报告生成。
摘要：
手术视频的自动摘要对于增强手术程序文档记录、支持手术训练以及促进术后分析至关重要。本文提出了一种位于人工智能与医学交叉领域的新方法，旨在开发可直接应用于真实世界手术场景的机器学习模型。我们提出了一个多模态框架，利用计算机视觉和大语言模型的最新进展来生成全面的视频摘要。该方法分为三个关键阶段。首先，将手术视频划分为多个片段，并使用视觉Transformer在帧级别提取视觉特征。此步骤侧重于检测工具、组织、器官和手术动作。其次，通过大语言模型将提取的特征转换为帧级描述。然后将这些描述与使用基于ViViT的编码器捕获的时间特征相结合，以生成反映每个视频片段更广泛上下文的片段级摘要。最后，使用专为摘要任务定制的专用LLM将片段级描述聚合为完整的手术报告。我们在CholecT50数据集上评估了该方法，使用了来自50个腹腔镜手术视频的器械和动作标注。结果显示出强大的性能，在工具检测方面达到96%的精确率，在时间上下文摘要方面达到0.74的BERT分数。这项工作有助于推进用于手术报告的AI辅助工具的发展，为更智能、更可靠的临床文档记录迈出了一步。
英文摘要：
The automatic summarization of surgical videos is essential for enhancing procedural documentation, supporting surgical training, and facilitating post-operative analysis. This paper presents a novel method at the intersection of artificial intelligence and medicine, aiming to develop machine learning models with direct real-world applications in surgical contexts. We propose a multi-modal framework that leverages recent advancements in computer vision and large language models to generate comprehensive video summaries. % The approach is structured in three key stages. First, surgical videos are divided into clips, and visual features are extracted at the frame level using visual transformers. This step focuses on detecting tools, tissues, organs, and surgical actions. Second, the extracted features are transformed into frame-level captions via large language models. These are then combined with temporal features, captured using a ViViT-based encoder, to produce clip-level summaries that reflect the broader context of each video segment. Finally, the clip-level descriptions are aggregated into a full surgical report using a dedicated LLM tailored for the summarization task. % We evaluate our method on the CholecT50 dataset, using instrument and action annotations from 50 laparoscopic videos. The results show strong performance, achieving 96\% precision in tool detection and a BERT score of 0.74 for temporal context summarization. This work contributes to the advancement of AI-assisted tools for surgical reporting, offering a step toward more intelligent and reliable clinical documentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
