arXiv 2606.158612026-06-16cs.CV

对象标记作为机器人手术中分割与视觉问答的桥梁

Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery

Department of Biomedical Engineering, Eindhoven University of Technology（埃因霍温理工大学生物医学工程系）；Department of Electrical Engineering, Eindhoven University of Technology（埃因霍温理工大学电气工程系）；Department of Surgery, University Medical Center Utrecht（乌得勒支大学医学中心外科）

Yiping Li, Ronald de Jong, Romy van Jaarsveld, Franco Badaloni, Gino Kuiper, Jelle Ruurda, Josien Pluim, Marcel Breeuwer

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出统一框架，联合像素级分割与视觉问答，通过VLM生成对象标记引导答案预测和分割掩码，在RAMIE和EndoVis18数据集上优于基线方法。

摘要：

机器人手术中的视觉问答（VQA），称为手术VQA，需要对复杂手术场景进行高级理解，并将视觉感知与语言推理相结合，具有支持手术培训和术中决策的潜力。最近的视觉-语言模型（VLM）通过参数高效微调显示出有希望的性能；然而，大多数现有方法依赖于粗粒度的视觉定位，通常仅限于边界框，这未能捕捉手术对象的细粒度空间结构。在这项工作中，我们提出了一个统一框架，在单个框架内联合执行像素级分割和视觉问答。我们的方法将VLM与基于Segment Anything Model（SAM）的解码器集成，并将场景元素表示为VLM生成的对象标记。这些对象标记指导答案预测，并进一步投影到基于SAM的解码器以产生分割掩码。通过分割和问答目标优化对象标记嵌入，模型学习空间基础表示，增强视觉推理，同时提供显式的像素级基础。我们在私有RAMIE（机器人辅助微创食管切除术）数据集和公共EndoVis18数据集上评估了所提出的方法，在手术VQA中始终优于基线方法。这些结果表明，将上下文感知的对象标记纳入视觉-语言模型可改善细粒度手术场景理解。

英文摘要：

Visual Question Answering (VQA) in robotic surgery, referred to as surgical VQA, requires high-level understanding of complex surgical scenes and the integration of visual perception with language reasoning, with the potential to support surgical training and intraoperative decision-making. Recent Vision-Language Models (VLMs) have shown promising performance through parameter-efficient fine-tuning; however, most existing approaches rely on coarse visual grounding, typically limited to bounding boxes, which fails to capture the fine-grained spatial structure of surgical objects. In this work, we propose a unified framework that jointly performs pixel-level segmentation and visual question answering within a single framework. Our approach integrates a VLM with a Segment Anything Model (SAM)-based decoder and represents scene elements as object tokens generated by the VLM. These object tokens guide answer prediction and are further projected to the SAM-based decoder to produce segmentation masks. By optimizing the object token embeddings through both segmentation and question answering objectives, the model learns spatially grounded representations that enhance visual reasoning while providing explicit pixel-level grounding. We evaluate the proposed method on the private RAMIE (Robot-Assisted Minimally Invasive Esophagectomy) dataset and the public EndoVis18 dataset, where it consistently outperforms baseline methods for surgical VQA. These results demonstrate that incorporating context-aware object tokens into vision-language models improves fine-grained surgical scene understanding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。