UIT-Saviors 在 MEDVQA-GI 2023：通过图像增强改进多模态学习用于胃肠道视觉问答
UIT-Saviors at MEDVQA-GI 2023: Improving Multimodal Learning with Image Enhancement for Gastrointestinal Visual Question Answering
AI总结：
针对胃肠道视觉问答任务，提出结合图像增强与多模态架构（BERT+视觉模型）的方法，实验证明基于Transformer的模型优于CNN，最佳方法在开发测试集上取得87.25%准确率和91.85% F1-Score。
摘要：
近年来，人工智能在医学和疾病诊断中发挥了重要作用，有许多应用值得一提，其中之一是医学视觉问答（Medical Visual Question Answering, MedVQA）。通过结合计算机视觉和自然语言处理，MedVQA系统可以协助专家根据给定问题从医学图像中提取相关信息，并提供精确的诊断答案。ImageCLEFmed-MEDVQA-GI-2023挑战赛在胃肠道领域开展了视觉问答任务，其中包括胃镜和肠镜图像。我们的团队通过提出一种带有图像增强的多模态学习方法来完成该挑战赛的任务1，以改善胃肠道图像上的VQA性能。该多模态架构采用BERT编码器以及基于卷积神经网络（CNN）和Transformer架构的不同预训练视觉模型，用于从问题和内窥镜图像中提取特征。本研究的结果突显了基于Transformer的视觉模型相对于CNN的优势，并证明了图像增强过程的有效性，八个视觉模型中有六个取得了更好的F1-Score。我们最佳的方法利用了BERT+BEiT融合和图像增强，在开发测试集上达到了87.25%的准确率和91.85%的F1-Score，同时在私有测试集上也取得了良好的结果，准确率为82.01%。
英文摘要：
In recent years, artificial intelligence has played an important role in medicine and disease diagnosis, with many applications to be mentioned, one of which is Medical Visual Question Answering (MedVQA). By combining computer vision and natural language processing, MedVQA systems can assist experts in extracting relevant information from medical image based on a given question and providing precise diagnostic answers. The ImageCLEFmed-MEDVQA-GI-2023 challenge carried out visual question answering task in the gastrointestinal domain, which includes gastroscopy and colonoscopy images. Our team approached Task 1 of the challenge by proposing a multimodal learning method with image enhancement to improve the VQA performance on gastrointestinal images. The multimodal architecture is set up with BERT encoder and different pre-trained vision models based on convolutional neural network (CNN) and Transformer architecture for features extraction from question and endoscopy image. The result of this study highlights the dominance of Transformer-based vision models over the CNNs and demonstrates the effectiveness of the image enhancement process, with six out of the eight vision models achieving better F1-Score. Our best method, which takes advantages of BERT+BEiT fusion and image enhancement, achieves up to 87.25% accuracy and 91.85% F1-Score on the development test set, while also producing good result on the private test set with accuracy of 82.01%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
