MedUnifier：利用离散视觉表示在医学数据上统一视觉-语言预训练与视觉生成任务
MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations
AI总结：
针对医学数据，提出统一视觉-语言预训练框架MedUnifier，通过视觉向量量化离散表示整合图像生成与多模态学习，在多项基准上达到最先进性能。
摘要：
尽管视觉-语言预训练（VLP）取得了显著进展，但当前方法主要强调特征提取和跨模态理解，对生成或转换视觉内容的关注有限。这一差距阻碍了模型从文本提示中合成连贯且新颖的视觉表示的能力，从而降低了多模态学习的有效性。在这项工作中，我们提出了MedUnifier，一个专为医学数据设计的统一VLP框架。MedUnifier无缝地将基于文本的图像生成能力与多模态学习策略（包括图像-文本对比对齐、图像-文本匹配和基于图像的文本生成）相结合。与依赖连续视觉表示的传统方法不同，我们的方法采用视觉向量量化，这不仅促进了更连贯的跨模态理解学习策略，还通过有效利用离散表示增强了多模态生成质量。我们框架的有效性通过在既有基准上的实验得到证明，包括单模态任务（监督微调）、跨模态任务（图像-文本检索和零样本图像分类）以及多模态任务（医学报告生成、图像合成），在这些任务中它实现了最先进的性能。MedUnifier还为医疗保健中的广泛语言和视觉任务提供了一个高度适应的工具，标志着朝着开发通用医学AI模型迈进的进步。
英文摘要：
Despite significant progress in Vision-Language Pre-training (VLP), current approaches predominantly emphasize feature extraction and cross-modal comprehension, with limited attention to generating or transforming visual content. This gap hinders the model's ability to synthesize coherent and novel visual representations from textual prompts, thereby reducing the effectiveness of multi-modal learning. In this work, we propose MedUnifier, a unified VLP framework tailored for medical data. MedUnifier seamlessly integrates text-grounded image generation capabilities with multi-modal learning strategies, including image-text contrastive alignment, image-text matching and image-grounded text generation. Unlike traditional methods that reply on continuous visual representations, our approach employs visual vector quantization, which not only facilitates a more cohesive learning strategy for cross-modal understanding but also enhances multi-modal generation quality by effectively leveraging discrete representations. Our framework's effectiveness is evidenced by the experiments on established benchmarks, including uni-modal tasks (supervised fine-tuning), cross-modal tasks (image-text retrieval and zero-shot image classification), and multi-modal tasks (medical report generation, image synthesis), where it achieves state-of-the-art performance across various tasks. MedUnifier also offers a highly adaptable tool for a wide range of language and vision tasks in healthcare, marking advancement toward the development of a generalizable AI model for medical applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
