CheXmix：用于医学影像的统一生成预训练
CheXmix: Unified Generative Pretraining for Vision Language Models in Medical Imaging
AI总结：
CheXmix通过统一早融合生成方法，在医学影像中实现更精确的联合表征学习，优于现有生成模型，在多个任务上表现突出。
摘要：
最近的医学多模态基础模型是通过将CLIP预训练的视觉编码器连接到语言模型中，采用LLaVA风格的微调方法构建多模态大语言模型（MLLMs）。这种两阶段、解耦的方法引入了投影层，可能扭曲视觉特征。这在医学影像中尤其令人担忧，因为细微的线索对准确诊断至关重要。相比之下，早期融合生成方法如Chameleon通过在单一统一序列中处理图像和文本标记，消除了投影瓶颈，实现了联合表征学习，利用了语言模型的归纳先验。我们提出了CheXmix，一种在大量胸部X光片与其放射学报告配对的语料库上训练的统一早融合生成模型。我们扩展了Chameleon的自回归框架，引入了两阶段多模态生成预训练策略，结合了掩码自动编码器的表示优势与MLLMs。所得到的模型高度灵活，支持在粗粒度和细粒度层面进行判别和生成任务。我们的方法在所有掩码比例上均比现有生成模型高出6.0%，在CheXpert分类任务中高图像掩码比例下超过CheXagent 8.6%。我们进一步在图像修复任务中比纯文本生成模型高出51.0%，在放射学报告生成的GREEN指标上超过CheXagent 45%。这些结果表明，CheXmix能够捕捉广泛胸片任务中的细粒度信息。我们的代码在：https://github.com/StanfordMIMI/CheXmix。
英文摘要：
Recent medical multimodal foundation models are built as multimodal LLMs (MLLMs) by connecting a CLIP-pretrained vision encoder to an LLM using LLaVA-style finetuning. This two-stage, decoupled approach introduces a projection layer that can distort visual features. This is especially concerning in medical imaging where subtle cues are essential for accurate diagnoses. In contrast, early-fusion generative approaches such as Chameleon eliminate the projection bottleneck by processing image and text tokens within a single unified sequence, enabling joint representation learning that leverages the inductive priors of language models. We present CheXmix, a unified early-fusion generative model trained on a large corpus of chest X-rays paired with radiology reports. We expand on Chameleon's autoregressive framework by introducing a two-stage multimodal generative pretraining strategy that combines the representational strengths of masked autoencoders with MLLMs. The resulting models are highly flexible, supporting both discriminative and generative tasks at both coarse and fine-grained scales. Our approach outperforms well-established generative models across all masking ratios by 6.0% and surpasses CheXagent by 8.6% on AUROC at high image masking ratios on the CheXpert classification task. We further inpaint images over 51.0% better than text-only generative models and outperform CheXagent by 45% on the GREEN metric for radiology report generation. These results demonstrate that CheXmix captures fine-grained information across a broad spectrum of chest X-ray tasks. Our code is at: https://github.com/StanfordMIMI/CheXmix.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
