BiomedJourney：基于多模态患者旅程指令学习的反事实生物医学图像生成
BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys
AI总结：
提出BiomedJourney方法，利用GPT-4处理多模态患者旅程生成指令三元组，训练潜在扩散模型进行反事实生物医学图像生成，并在MIMIC-CXR数据集上显著优于InstructPix2Pix等方法。
摘要：
以 InstructPix2Pix 为代表，基于自然语言指令的图像编辑指令学习已取得快速进展。在生物医学领域，此类方法可应用于反事实图像生成，有助于区分因果结构与虚假相关性，并促进疾病进展建模的鲁棒图像解释。然而，通用图像编辑模型不适用于生物医学领域，且反事实生物医学图像生成在很大程度上尚未被充分探索。在本文中，我们提出了 BiomedJourney，一种通过从多模态患者旅程中进行指令学习来生成反事实生物医学图像的新方法。给定一名患者在不同时间点拍摄的两张生物医学图像，我们使用 GPT-4 处理相应的成像报告，并生成关于疾病进展的自然语言描述。由此得到的三元组（先前图像，进展描述，新图像）随后被用于训练一个潜在扩散模型以进行反事实生物医学图像生成。鉴于图像时间序列数据的相对稀缺性，我们引入了一种两阶段课程学习：首先使用更为丰富的单图像-报告对（带有虚拟先前图像）预训练去噪网络，然后使用反事实三元组继续训练。使用标准 MIMIC-CXR 数据集进行的实验证明了我们方法的潜力。在针对反事实医学图像生成的一系列全面测试中，BiomedJourney 大幅优于先前在指令图像编辑和医学图像生成领域的最先进方法，如 InstructPix2Pix 和 RoentGen。为了促进未来在反事实医学生成方面的研究，我们计划发布我们的指令学习代码和预训练模型。
英文摘要：
Rapid progress has been made in instruction-learning for image editing with natural-language instruction, as exemplified by InstructPix2Pix. In biomedicine, such methods can be applied to counterfactual image generation, which helps differentiate causal structure from spurious correlation and facilitate robust image interpretation for disease progression modeling. However, generic image-editing models are ill-suited for the biomedical domain, and counterfactual biomedical image generation is largely underexplored. In this paper, we present BiomedJourney, a novel method for counterfactual biomedical image generation by instruction-learning from multimodal patient journeys. Given a patient with two biomedical images taken at different time points, we use GPT-4 to process the corresponding imaging reports and generate a natural language description of disease progression. The resulting triples (prior image, progression description, new image) are then used to train a latent diffusion model for counterfactual biomedical image generation. Given the relative scarcity of image time series data, we introduce a two-stage curriculum that first pretrains the denoising network using the much more abundant single image-report pairs (with dummy prior image), and then continues training using the counterfactual triples. Experiments using the standard MIMIC-CXR dataset demonstrate the promise of our method. In a comprehensive battery of tests on counterfactual medical image generation, BiomedJourney substantially outperforms prior state-of-the-art methods in instruction image editing and medical image generation such as InstructPix2Pix and RoentGen. To facilitate future study in counterfactual medical generation, we plan to release our instruction-learning code and pretrained models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
