用于医学视觉问答的具有单模态和多模态对比损失的掩码视觉与语言预训练
Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answering
AI总结：
本文提出一种结合单模态与多模态对比损失及掩码语言建模的医学视觉问答自监督预训练方法，在三个公开数据集上取得最先进性能，准确率分别提升2.2%、14.7%和1.7%。
摘要：
医学视觉问答（VQA）是一项具有挑战性的任务，要求同时考虑视觉和语言信息来回答给定医学图像的临床问题。然而，由于医学VQA训练数据规模较小，预训练-微调范式已成为提高模型泛化性能的常用解决方案。本文提出了一种新颖的自监督方法，利用医学图像描述数据集学习输入图像和文本的单模态及多模态特征表示，同时利用单模态和多模态对比损失，并以掩码语言建模和图像文本匹配作为预训练目标。然后将预训练模型迁移到下游医学VQA任务。所提出的方法在三个公开可用的医学VQA数据集上取得了最先进（SOTA）性能，准确率分别显著提高了2.2%、14.7%和1.7%。此外，我们进行了全面分析，以验证该方法不同组件的有效性，并研究了不同的预训练设置。我们的代码和模型可在https://github.com/pengfeiliHEU/MUMC获取。
英文摘要：
Medical visual question answering (VQA) is a challenging task that requires answering clinical questions of a given medical image, by taking consider of both visual and language information. However, due to the small scale of training data for medical VQA, pre-training fine-tuning paradigms have been a commonly used solution to improve model generalization performance. In this paper, we present a novel self-supervised approach that learns unimodal and multimodal feature representations of input images and text using medical image caption datasets, by leveraging both unimodal and multimodal contrastive losses, along with masked language modeling and image text matching as pretraining objectives. The pre-trained model is then transferred to downstream medical VQA tasks. The proposed approach achieves state-of-the-art (SOTA) performance on three publicly available medical VQA datasets with significant accuracy improvements of 2.2%, 14.7%, and 1.7% respectively. Besides, we conduct a comprehensive analysis to validate the effectiveness of different components of the approach and study different pre-training settings. Our codes and models are available at https://github.com/pengfeiliHEU/MUMC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
