arXiv 2506.194692025-06-24cs.CVcs.AI

Surgery-R1：通过强化学习推进外科视觉问答定位回答的推理多模态大语言模型

Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learning

Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Department of Thoracic Surgery, the Seventh Affiliated Hospital, Sun Yat-sen University（中山大学第七附属医院胸外科部门）；Bioengineering/Imperial-X, Imperial College London（生物工程/Imperial-X，帝国理工学院伦敦分校）；ROAS Thrust, Hong Kong University of Science and Technology (Guangzhou)（ROAS项目，香港科技大学（广州））；Department of Electronic and Computer Engineering, Hong Kong SAR（香港特别行政区电子与计算机工程系）

Pengfei Hao, Shuaibo Li, Hongqiu Wang, Zhizhuo Kou, Junhang Zhang, Guang Yang, Lei Zhu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对外科手术视觉问答定位回答缺乏推理能力的问题，构建Surgery-R1-54k数据集，提出基于两阶段微调和多模态一致性奖励的推理多模态大模型Surgery-R1，显著优于现有方法。

摘要：

近年来，外科场景理解领域取得了显著进展，特别是在机器人手术中的视觉问答定位回答（Surgical-VQLA）任务上。然而，现有的Surgical-VQLA模型缺乏深层推理能力和可解释性，这限制了其在临床应用中的可靠性和发展潜力。为解决这一问题，受推理多模态大语言模型（MLLMs）发展的启发，我们首先构建了Surgery-R1-54k数据集，包括视觉问答（Visual-QA）、定位问答（Grounding-QA）和思维链（CoT）的配对数据。然后，我们提出了首个用于Surgical-VQLA的推理多模态大语言模型（Surgery-R1）。在我们的Surgery-R1中，我们设计了一个两阶段微调机制，通过利用监督微调（SFT）和强化微调（RFT）使基础MLLM具备复杂推理能力。此外，为了在我们的RFT中实现高效且高质量的基于规则的奖励系统，我们设计了一种多模态一致性奖励机制，以缓解手术场景中可能出现的定位幻觉。实验结果表明，Surgery-R1在Surgical-VQLA任务上优于其他现有的最先进（SOTA）模型和广泛使用的MLLMs，同时也验证了其推理能力和我们方法的有效性。代码和数据集将整理在https://github.com/FiFi-HAO467/Surgery-R1。

英文摘要：

In recent years, significant progress has been made in the field of surgical scene understanding, particularly in the task of Visual Question Localized-Answering in robotic surgery (Surgical-VQLA). However, existing Surgical-VQLA models lack deep reasoning capabilities and interpretability in surgical scenes, which limits their reliability and potential for development in clinical applications. To address this issue, inspired by the development of Reasoning Multimodal Large Language Models (MLLMs), we first build the Surgery-R1-54k dataset, including paired data for Visual-QA, Grounding-QA, and Chain-of-Thought (CoT). Then, we propose the first Reasoning MLLM for Surgical-VQLA (Surgery-R1). In our Surgery-R1, we design a two-stage fine-tuning mechanism to enable the basic MLLM with complex reasoning abilities by utilizing supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). Furthermore, for an efficient and high-quality rule-based reward system in our RFT, we design a Multimodal Coherence reward mechanism to mitigate positional illusions that may arise in surgical scenarios. Experiment results demonstrate that Surgery-R1 outperforms other existing state-of-the-art (SOTA) models in the Surgical-VQLA task and widely-used MLLMs, while also validating its reasoning capabilities and the effectiveness of our approach. The code and dataset will be organized in https://github.com/FiFi-HAO467/Surgery-R1.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。