arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.03231cs.LGcs.AIcs.CVcs.HCeess.AS
收藏

用于手术反馈分类的深度多模态融合

Deep Multimodal Fusion for Surgical Feedback Classification

California Institute of Technology(加州理工学院);University of Southern California(南加州大学);NVIDIA(英伟达);Cedars-Sinai Medical Center(西达赛奈医疗中心)

Rafal Kocielnik, Elyssa Y. Wong, Timothy N. Chu, Lydia Lin, De-An Huang, Jiayun Wang, Anima Anandkumar, Andrew J. Hung

AI总结:

本文提出融合文本、音频与视频的深度多标签模型,对五类实时手术反馈自动分类,并验证分阶段训练与高质量转录的性能增益。

摘要:

量化经验丰富的外科医生在手术过程中向受训者提供的实时非正式反馈,对于外科训练中的技能提升十分重要。此类在实时手术室中的反馈本质上是多模态的,既包括言语对话(例如问答),也包括非言语元素(例如通过指向解剖结构等视觉线索)。在本研究中,我们采用经临床验证的五类手术反馈分类:“Anatomic(解剖)”、“Technical(技术)”、“Procedural(流程)”、“Praise(表扬)”和“Visual Aid(视觉辅助)”。随后,我们开发了一个多标签机器学习模型,从文本、音频和视频模态输入中对这五类手术反馈进行分类。我们工作的最终目标是帮助大规模自动化标注实时情境化手术反馈。我们的手术反馈自动分类取得了71.5至77.6的AUC,融合使性能提升了3.1%。我们还表明,由专家对反馈音频进行高质量人工转录可将AUC提升至76.5至96.2之间,这展示了一条清晰的未来改进路径。根据实验,我们发现Staged训练策略(首先分别预训练每种模态,然后联合训练它们)比将不同模态放在一起训练更有效。我们还给出了关于不同反馈类别中各模态重要性的直观发现。本工作首次重要考察了基于文本、音频和视频模态对真实世界实时手术反馈进行自动分类的可行性。

英文摘要:

Quantification of real-time informal feedback delivered by an experienced surgeon to a trainee during surgery is important for skill improvements in surgical training. Such feedback in the live operating room is inherently multimodal, consisting of verbal conversations (e.g., questions and answers) as well as non-verbal elements (e.g., through visual cues like pointing to anatomic elements). In this work, we leverage a clinically-validated five-category classification of surgical feedback: "Anatomic", "Technical", "Procedural", "Praise" and "Visual Aid". We then develop a multi-label machine learning model to classify these five categories of surgical feedback from inputs of text, audio, and video modalities. The ultimate goal of our work is to help automate the annotation of real-time contextual surgical feedback at scale. Our automated classification of surgical feedback achieves AUCs ranging from 71.5 to 77.6 with the fusion improving performance by 3.1%. We also show that high-quality manual transcriptions of feedback audio from experts improve AUCs to between 76.5 and 96.2, which demonstrates a clear path toward future improvements. Empirically, we find that the Staged training strategy, with first pre-training each modality separately and then training them jointly, is more effective than training different modalities altogether. We also present intuitive findings on the importance of modalities for different feedback categories. This work offers an important first look at the feasibility of automated classification of real-world live surgical feedback based on text, audio, and video modalities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑