机器人辅助手术中的实时多模态活动感知错误检测
Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery
AI总结：
提出统一框架，利用视频、运动学数据和文本提示等多模态输入，通过活动提示和活动感知视觉嵌入，在JIGSAWS和SAR-RARP50数据集上分别提升F1分数达5%和16.6%。
摘要：
机器人辅助微创手术提高了手术精度，但也引入了复杂性，因此技术错误检测对于确保患者安全至关重要。当前使用视频数据的执行错误检测方法往往忽略了手术程序层次结构中活动和错误类型的细粒度上下文描述，并且未能充分利用互补的多模态信息。我们提出了一个统一的执行错误检测框架，利用多模态输入，包括视频、运动学数据和描述性文本提示。通过活动提示，我们将描述性语言整合到手势级活动、器械-物体交互和错误定义中。我们还引入了从手术活动标签预训练的视觉编码器导出的活动感知视觉嵌入，以比较对比语言-图像嵌入与传统基于图像的嵌入在错误检测中的有效性。通过将运动学数据与视频和文本模态无缝集成，我们的框架显著提高了错误检测性能。在JIGSAWS和SAR-RARP50数据集上，相比最先进的基线，我们分别实现了高达5%和16.6%的F1分数提升，证明了将精心设计的文本提示与多模态数据相结合对于准确错误检测的价值。
英文摘要：
Robot-assisted minimally invasive surgery improves surgical precision but introduces complexity, making technical error detection essential for ensuring patient safety. Current executional error detection methods using video data often overlook fine-grained contextual descriptions of activities and error types within the hierarchical structure of surgical procedures. They also under-utilize complementary multimodal information. We propose a unified framework for executional error detection that leverages multimodal input, including video, kinematics, and descriptive textual prompts. Through activity prompting, we integrate descriptive language in gesture-level activities, instrument-object interactions, and error definitions. We also introduce activity-aware visual embeddings derived from vision encoders pretrained on surgical activity labels to compare the effectiveness of contrastive language-image embeddings with traditional image-based embeddings for error detection. By seamlessly integrating kinematic data with video and textual modalities, our framework significantly improves error detection performance. Achieving up to 5\% and 16.6\% F1 score improvements over state-of-the-art baselines on the JIGSAWS and SAR-RARP50 datasets, respectively, we demonstrate the value of combining curated textual prompts with multimodal data for accurate error detection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
