用生成的放射学报告为胸部X光视觉问答提供依据
Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reports
AI总结：
提出一种利用生成的放射学报告作为依据的胸部X光视觉问答方法，通过报告生成与答案生成两步流程，同时处理单图像和图像差异问题，并在Medical-Diff-VQA数据集上取得最先进性能。
摘要：
我们提出了一种针对胸部X光（CXR）视觉问答（VQA）的新方法，同时处理单图像和图像差异问题。单图像问题关注特定CXR中的异常（“图像X中可见哪些异常？”），而图像差异问题则比较在不同时间点获取的两张纵向CXR（“图像X和Y之间有哪些差异？”）。我们进一步探讨了将放射学报告整合进来如何能提升VQA模型的性能。虽然先前的方法已经证明了放射学报告在预训练阶段的作用，但我们扩展了这一想法，表明这些报告也可以作为额外输入来改进VQA模型的预测答案。首先，我们提出了一种统一的方法，该方法能处理两类问题并自回归地生成答案。对于单图像问题，模型接收一张CXR。对于图像差异问题，模型接收同一患者在不同时间点获取的两张CXR，使模型能够检测并描述时间上的变化。受“思维链推理”的启发，我们证明通过用针对同一CXR预测的放射学报告来为答案生成模块提供依据，可以提高CXR VQA任务的性能。在我们的方法中，VQA模型分为两个步骤：i) 报告生成（RG）和 ii) 答案生成（AG）。我们的结果表明，将预测的放射学报告作为证据纳入AG模型，能提升单图像和图像差异问题的性能，并在Medical-Diff-VQA数据集上取得了最先进的结果。
英文摘要：
We present a novel approach to Chest X-ray (CXR) Visual Question Answering (VQA), addressing both single-image image-difference questions. Single-image questions focus on abnormalities within a specific CXR ("What abnormalities are seen in image X?"), while image-difference questions compare two longitudinal CXRs acquired at different time points ("What are the differences between image X and Y?"). We further explore how the integration of radiology reports can enhance the performance of VQA models. While previous approaches have demonstrated the utility of radiology reports during the pre-training phase, we extend this idea by showing that the reports can also be leveraged as additional input to improve the VQA model's predicted answers. First, we propose a unified method that handles both types of questions and auto-regressively generates the answers. For single-image questions, the model is provided with a single CXR. For image-difference questions, the model is provided with two CXRs from the same patient, captured at different time points, enabling the model to detect and describe temporal changes. Taking inspiration from 'Chain-of-Thought reasoning', we demonstrate that performance on the CXR VQA task can be improved by grounding the answer generator module with a radiology report predicted for the same CXR. In our approach, the VQA model is divided into two steps: i) Report Generation (RG) and ii) Answer Generation (AG). Our results demonstrate that incorporating predicted radiology reports as evidence to the AG model enhances performance on both single-image and image-difference questions, achieving state-of-the-art results on the Medical-Diff-VQA dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
