arXiv 2408.049582024-09-01cs.CVcs.RO

Surgical-VQLA++：用于机器人手术中校准鲁棒视觉问答定位的对抗对比学习

Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery

The Chinese University of Hong Kong（香港中文大学）；Shenzhen Research Institute, The Chinese University of Hong Kong（香港中文大学深圳研究院）；University College London（伦敦大学学院）；National University of Singapore（新加坡国立大学）

Long Bai, Guankun Wang, Mobarakol Islam, Lalithkumar Seenivasan, An Wang, Hongliang Ren

备注：: Accepted by Information Fusion. Code and data availability: https://github.com/longbai1006/Surgical-VQLAPlus

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Surgical-VQLA++方法，通过校准协同注意力门控视觉-语言嵌入和对抗对比学习，实现手术场景中鲁棒的视觉问答定位，并扩展了手术VQLA数据集。

摘要：

医学视觉问答（VQA）弥合了视觉信息与临床决策之间的差距，使医生能够从临床图像和视频中提取理解。特别是，手术VQA可以增强对手术数据的解读，有助于准确诊断、有效教育和临床干预。然而，VQA模型无法在视觉上指示与所提问题对应的感兴趣区域，导致对手术场景的理解不完整。为了解决这一问题，我们提出了手术视觉问答定位（VQLA）方法，用于对手术图像的具体查询做出精确且上下文感知的回答。此外，为了应对手术场景中对安全性的强烈需求以及图像采集和传输中可能出现的损坏，我们提出了一种名为校准协同注意力门控视觉-语言（C$^2$G-ViL）嵌入的新方法，以有效地整合和对齐多模态信息。另外，我们利用基于对抗样本的对比学习策略来提升性能和鲁棒性。我们还扩展了EndoVis-18-VQLA和EndoVis-17-VQLA数据集，以拓宽数据的范围和应用。在上述数据集上的大量实验证明了我们解决方案的卓越性能和鲁棒性。我们的解决方案能够有效对抗现实世界中的图像损坏。因此，我们提出的方法可以作为辅助手术教育、患者护理和改善手术结果的有效工具。

英文摘要：

Medical visual question answering (VQA) bridges the gap between visual information and clinical decision-making, enabling doctors to extract understanding from clinical images and videos. In particular, surgical VQA can enhance the interpretation of surgical data, aiding in accurate diagnoses, effective education, and clinical interventions. However, the inability of VQA models to visually indicate the regions of interest corresponding to the given questions results in incomplete comprehension of the surgical scene. To tackle this, we propose the surgical visual question localized-answering (VQLA) for precise and context-aware responses to specific queries regarding surgical images. Furthermore, to address the strong demand for safety in surgical scenarios and potential corruptions in image acquisition and transmission, we propose a novel approach called Calibrated Co-Attention Gated Vision-Language (C$^2$G-ViL) embedding to integrate and align multimodal information effectively. Additionally, we leverage the adversarial sample-based contrastive learning strategy to boost our performance and robustness. We also extend our EndoVis-18-VQLA and EndoVis-17-VQLA datasets to broaden the scope and application of our data. Extensive experiments on the aforementioned datasets demonstrate the remarkable performance and robustness of our solution. Our solution can effectively combat real-world image corruption. Thus, our proposed approach can serve as an effective tool for assisting surgical education, patient care, and enhancing surgical outcomes.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。