arXiv 2511.033252025-11-06cs.CV

SurgViVQA：面向手术场景理解的时间定位视频问答

SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding

Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB)（电子、信息与生物工程系）；Politecnico di Milano（米兰理工大学）；IRCCS Humanitas Research Hospital（IRCCS人类itas研究医院）；UCL Hawkes Institute and Department of Computer Science（UCL Hawkes研究所和计算机科学系）；University College London（伦敦大学学院）；University of Manchester（曼彻斯特大学）

Mauro Orazio Drago, Luca Carlini, Pelinsu Celebi Balyemez, Dennis Pierantozzi, Chiara Lena, Cesare Hassan, Danail Stoyanov, Elena De Momi, Sophia Bano, Mobarak I. Hoque

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出手术视频问答模型SurgViVQA，通过掩码视频-文本编码器融合视频与问题特征并由大语言模型解码，结合新构建的REAL-Colon-VQA数据集，在关键词准确率上显著超越现有图像基准模型。

摘要：

手术领域的视频问答（VideoQA）旨在通过使AI模型能够对时间上连贯的事件而非孤立帧进行推理，来增强术中理解。当前方法仅限于静态图像特征，且现有数据集往往缺乏时间标注，忽略了对于准确程序解读至关重要的动态信息。我们提出SurgViVQA，一个将视觉推理从静态图像扩展到动态手术场景的手术视频问答模型。该模型使用掩码视频-文本编码器（Masked Video--Text Encoder）融合视频和问题特征，捕捉运动以及工具-组织交互等时间线索，随后由微调后的大语言模型（LLM）将其解码为连贯的答案。为评估其性能，我们构建了REAL-Colon-VQA，一个结肠镜视频数据集，包含运动相关问题和诊断属性，以及带有改写或语义变化表述的模板外问题，用于评估模型的鲁棒性。在REAL-Colon-VQA和公开的EndoVis18-VQA数据集上的实验验证表明，SurgViVQA优于现有的基于图像的VQA基准模型，尤其在关键词准确率方面，在REAL-Colon-VQA上比PitVQA提高了+11%，在EndoVis18-VQA上提高了+9%。对问题的扰动研究进一步证实了其对问题表述变化的泛化能力和鲁棒性有所提升。SurgViVQA和REAL-Colon-VQA数据集为手术视频问答中的时间感知理解提供了一个框架，使AI模型能够更有效地解读动态程序上下文。代码和数据集可在https://github.com/madratak/SurgViVQA获取。

英文摘要：

Video Question Answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events rather than isolated frames. Current approaches are limited to static image features, and available datasets often lack temporal annotations, ignoring the dynamics critical for accurate procedural interpretation. We propose SurgViVQA, a surgical VideoQA model that extends visual reasoning from static images to dynamic surgical scenes. It uses a Masked Video--Text Encoder to fuse video and question features, capturing temporal cues such as motion and tool--tissue interactions, which a fine-tuned large language model (LLM) then decodes into coherent answers. To evaluate its performance, we curated REAL-Colon-VQA, a colonoscopic video dataset that includes motion-related questions and diagnostic attributes, as well as out-of-template questions with rephrased or semantically altered formulations to assess model robustness. Experimental validation on REAL-Colon-VQA and the public EndoVis18-VQA dataset shows that SurgViVQA outperforms existing image-based VQA benchmark models, particularly in keyword accuracy, improving over PitVQA by +11\% on REAL-Colon-VQA and +9\% on EndoVis18-VQA. A perturbation study on the questions further confirms improved generalizability and robustness to variations in question phrasing. SurgViVQA and the REAL-Colon-VQA dataset provide a framework for temporally-aware understanding in surgical VideoQA, enabling AI models to interpret dynamic procedural contexts more effectively. Code and dataset available at https://github.com/madratak/SurgViVQA.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。