arXiv 2304.099742023-07-22cs.CVcs.AIeess.IV

SurgicalGPT：用于手术领域视觉问答的端到端语言-视觉GPT

SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery

National University of Singapore（新加坡国立大学）；University College London（伦敦大学学院）；National Institute of Technology, Tiruchirappalli（蒂鲁吉拉伯利国立理工学院）；Chinese University of Hong Kong（香港中文大学）；Shun Hing Institute of Advanced Engineering, Chinese University of Hong Kong（香港中文大学信兴高等工程研究所）

Lalithkumar Seenivasan, Mobarakol Islam, Gokul Kannan, Hongliang Ren

备注：: The manuscript is accepted in MICCAI 2023. Code are available at: https://github.com/lalithjets/SurgicalGPT

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究针对GPT无法原生处理视觉信息的问题，提出可端到端训练的LV-GPT模型，通过扩展GPT2加入视觉模块并优化令牌排序，在多个手术VQA数据集上性能超越现有最优模型，还补充了问题类型标注并分析了相关模块影响。

摘要：

基于GPT的大语言模型（LLM）的发展正在革新自然语言处理领域，其在各领域的应用呈指数级增长。这类自回归大语言模型采用单向注意力机制，能够生成连贯的长段落。然而，对于需要同时处理视觉和语言信息的视觉问答（VQA）任务，通常会使用具备双向注意力的模型或采用融合技术的模型，以一次性捕获多模态的上下文信息。由于GPT本身不具备处理视觉令牌的能力，为了将GPT模型的技术进展应用于机器人手术领域的视觉问答任务，我们设计了一种可端到端训练的语言-视觉GPT（LV-GPT）模型，该模型在GPT2模型基础上扩展，新增了视觉输入（图像）能力。所提出的LV-GPT包含一个特征提取器（视觉令牌化器）以及视觉令牌嵌入层（令牌类型嵌入与位置嵌入）。考虑到GPT模型中单向注意力的局限性及其生成连贯长段落的能力，我们精心将文本令牌排列在视觉令牌之前，模拟人类先理解问题、再从图像中推导答案的思考过程。在量化实验方面，我们证明LV-GPT模型在两个公开可用的手术视觉问答数据集（基于内窥镜视觉挑战赛2018年机器人场景分割数据集与CholecTriplet2021数据集构建）以及我们新标注的数据集（基于整体手术场景数据集构建）上，性能均优于其他最先进的VQA模型。我们还对三个数据集进行了进一步标注，新增了问题类型标注，以支持子类型分析。此外，我们还深入研究并展示了LV-GPT模型中令牌排序、视觉令牌的类型嵌入与位置嵌入带来的效果影响。

英文摘要：

Advances in GPT-based large language models (LLMs) are revolutionizing natural language processing, exponentially increasing its use across various domains. Incorporating uni-directional attention, these autoregressive LLMs can generate long and coherent paragraphs. However, for visual question answering (VQA) tasks that require both vision and language processing, models with bi-directional attention or models employing fusion techniques are often employed to capture the context of multiple modalities all at once. As GPT does not natively process vision tokens, to exploit the advancements in GPT models for VQA in robotic surgery, we design an end-to-end trainable Language-Vision GPT (LV-GPT) model that expands the GPT2 model to include vision input (image). The proposed LV-GPT incorporates a feature extractor (vision tokenizer) and vision token embedding (token type and pose). Given the limitations of unidirectional attention in GPT models and their ability to generate coherent long paragraphs, we carefully sequence the word tokens before vision tokens, mimicking the human thought process of understanding the question to infer an answer from an image. Quantitatively, we prove that the LV-GPT model outperforms other state-of-the-art VQA models on two publically available surgical-VQA datasets (based on endoscopic vision challenge robotic scene segmentation 2018 and CholecTriplet2021) and on our newly annotated dataset (based on the holistic surgical scene dataset). We further annotate all three datasets to include question-type annotations to allow sub-type analysis. Furthermore, we extensively study and present the effects of token sequencing, token type and pose embedding for vision tokens in the LV-GPT model.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。