面向纵向胸部X光片差异视觉问答的视觉语言模型预训练
Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays
AI总结：
针对纵向胸部X光差异视觉问答任务，提出分步预训练的新型视觉语言模型PLURAL，在相关任务上均优于现有最优方法。
摘要：
差异视觉问答（diff-VQA）是一项具有挑战性的任务，要求基于一对图像之间的差异回答复杂问题。该任务在胸部X光片阅片中尤为重要，因为放射科医生在临床实践中经常会比较同一名患者在不同时间拍摄的多张图像，以追踪疾病进展和严重程度的变化。然而，以往的研究专注于为diff-VQA任务设计特定的网络架构，错失了利用预训练视觉语言模型（VLM）提升模型性能的机会。本文提出了一种名为PLURAL的新型VLM，该模型针对diff-VQA任务，在自然图像数据和纵向胸部X光数据上进行预训练。模型采用分步方法开发：首先在自然图像和文本上进行预训练，随后使用纵向胸部X光数据训练。纵向数据包含成对的X光图像，以及问答集和描述肺部异常与疾病随时间变化的放射科医生报告。我们的实验结果表明，PLURAL模型不仅在纵向X光的diff-VQA任务上优于现有最优方法，在单张X光图像的常规VQA任务中也表现更优。通过大量实验，我们证明了所提出的VLM架构和预训练方法在提升模型性能方面的有效性。
英文摘要：
Difference visual question answering (diff-VQA) is a challenging task that requires answering complex questions based on differences between a pair of images. This task is particularly important in reading chest X-ray images because radiologists often compare multiple images of the same patient taken at different times to track disease progression and changes in its severity in their clinical practice. However, previous works focused on designing specific network architectures for the diff-VQA task, missing opportunities to enhance the model's performance using a pretrained vision-language model (VLM). Here, we introduce a novel VLM called PLURAL, which is pretrained on natural and longitudinal chest X-ray data for the diff-VQA task. The model is developed using a step-by-step approach, starting with being pretrained on natural images and texts, followed by being trained using longitudinal chest X-ray data. The longitudinal data consist of pairs of X-ray images, along with question-answer sets and radiologist's reports that describe the changes in lung abnormalities and diseases over time. Our experimental results show that the PLURAL model outperforms state-of-the-art methods not only in diff-VQA for longitudinal X-rays but also in conventional VQA for a single X-ray image. Through extensive experiments, we demonstrate the effectiveness of the proposed VLM architecture and pretraining method in improving the model's performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
