PitVQA：用于垂体手术视觉问答的图像接地文本嵌入LLM
PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery
AI总结：
本文提出PitVQA数据集和PitVQA-Net模型，通过图像接地文本嵌入增强GPT2，用于垂体手术VQA，在PitVQA和EndoVis18-VQA上分别提升平衡准确率8%和9%。
摘要：
在外科领域，利用大型语言模型（LLMs）进行视觉问答（VQA）为改善术中决策和促进直观的外科医生与人工智能交互提供了独特机会。然而，用于外科VQA的LLMs开发受到缺乏包含复杂推理任务的多样且广泛数据集的阻碍。此外，由于图像和文本模态之间的固有差异以及对齐它们的复杂性，图像和文本模态的上下文融合仍然是一个开放的研究挑战。本文介绍了PitVQA，一个专门为内镜经鼻垂体手术VQA设计的新数据集，以及PitVQA-Net，一种基于GPT2的适应，具有新颖的图像接地文本嵌入用于外科VQA。PitVQA包含25个手术过程视频和丰富的问题-答案对集合，涵盖关键手术方面，如阶段和步骤识别、上下文理解、工具检测和定位以及工具-组织交互。PitVQA-Net包括一种新颖的图像接地文本嵌入，将图像和文本特征投影到共享嵌入空间，以及带有激励块分类头的GPT2骨干，以在内镜经鼻垂体手术的复杂领域内生成上下文相关的答案。我们的图像接地文本嵌入利用联合嵌入、交叉注意力和上下文表示来理解问题与手术图像之间的上下文关系。我们在PitVQA和公开可用的EndoVis18-VQA数据集上展示了PitVQA-Net的有效性，与最新基线相比，平衡准确率分别提高了8%和9%。我们的代码和数据集可在https://github.com/mobarakol/PitVQA获取。
英文摘要：
Visual Question Answering (VQA) within the surgical domain, utilizing Large Language Models (LLMs), offers a distinct opportunity to improve intra-operative decision-making and facilitate intuitive surgeon-AI interaction. However, the development of LLMs for surgical VQA is hindered by the scarcity of diverse and extensive datasets with complex reasoning tasks. Moreover, contextual fusion of the image and text modalities remains an open research challenge due to the inherent differences between these two types of information and the complexity involved in aligning them. This paper introduces PitVQA, a novel dataset specifically designed for VQA in endonasal pituitary surgery and PitVQA-Net, an adaptation of the GPT2 with a novel image-grounded text embedding for surgical VQA. PitVQA comprises 25 procedural videos and a rich collection of question-answer pairs spanning crucial surgical aspects such as phase and step recognition, context understanding, tool detection and localization, and tool-tissue interactions. PitVQA-Net consists of a novel image-grounded text embedding that projects image and text features into a shared embedding space and GPT2 Backbone with an excitation block classification head to generate contextually relevant answers within the complex domain of endonasal pituitary surgery. Our image-grounded text embedding leverages joint embedding, cross-attention and contextual representation to understand the contextual relationship between questions and surgical images. We demonstrate the effectiveness of PitVQA-Net on both the PitVQA and the publicly available EndoVis18-VQA dataset, achieving improvements in balanced accuracy of 8% and 9% over the most recent baselines, respectively. Our code and dataset is available at https://github.com/mobarakol/PitVQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
