arXiv 2405.109482025-03-16cs.CVcs.AIcs.ROeess.IV

Surgical-LVLM：学习适配大型视觉-语言模型以用于机器人手术中的扎根视觉问答

Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery

The Chinese University of Hong Kong（香港中文大学）；Universiti Malaya（马来亚大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港智能系统研究所人工智能与机器人中心）；University College London（伦敦大学学院）

Guankun Wang, Long Bai, Wan Jun Nah, Jie Wang, Zhaoxi Zhang, Zhen Chen, Jinlin Wu, Mobarakol Islam, Hongbin Liu, Hongliang Ren

备注：: The manuscript is accepted by ICLR 2025 FM-Wild Workshop

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出面向复杂手术场景的个性化大型视觉-语言模型 Surgical-LVLM，通过 VP-LoRA 与 TIT 模块增强多模态理解和视觉定位，在多个 EndoVis 基准上取得最优性能。

摘要：

近期，外科视觉问答（Surgical-VQA）及相关区域定位的进展，在机器人和医疗应用中展现出巨大前景，回应了个性化手术指导对自动化方法的迫切需求。然而，现有模型主要提供简单的结构化答案，并且由于识别长程依赖和对齐多模态信息的能力有限，在复杂场景中表现困难。本文提出 Surgical-LVLM，这是一种专为复杂手术场景定制的新型个性化大型视觉-语言模型。借助预训练的大型视觉-语言模型和专门的 Visual Perception LoRA（VP-LoRA）模块，该模型擅长理解手术语境中的复杂视觉-语言任务。为解决视觉定位任务，我们提出 Token-Interaction（TIT）模块，在将大型视觉-语言模型（LVLM）的语言响应投影到潜在空间后，增强定位模块与这些语言响应之间的交互。我们在多个基准上证明了 Surgical-LVLM 的有效性，包括 EndoVis-17-VQLA、EndoVis-18-VQLA，以及新引入的 EndoVis Conversations 数据集；该模型树立了新的性能标准。本工作通过提供一种具备情境感知能力的解决方案，为推进自动化手术指导领域作出贡献。

英文摘要：

Recent advancements in Surgical Visual Question Answering (Surgical-VQA) and related region grounding have shown great promise for robotic and medical applications, addressing the critical need for automated methods in personalized surgical mentorship. However, existing models primarily provide simple structured answers and struggle with complex scenarios due to their limited capability in recognizing long-range dependencies and aligning multimodal information. In this paper, we introduce Surgical-LVLM, a novel personalized large vision-language model tailored for complex surgical scenarios. Leveraging the pre-trained large vision-language model and specialized Visual Perception LoRA (VP-LoRA) blocks, our model excels in understanding complex visual-language tasks within surgical contexts. In addressing the visual grounding task, we propose the Token-Interaction (TIT) module, which strengthens the interaction between the grounding module and the language responses of the Large Visual Language Model (LVLM) after projecting them into the latent space. We demonstrate the effectiveness of Surgical-LVLM on several benchmarks, including EndoVis-17-VQLA, EndoVis-18-VQLA, and a newly introduced EndoVis Conversations dataset, which sets new performance standards. Our work contributes to advancing the field of automated surgical mentorship by providing a context-aware solution.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。