EMIXER：通过自我监控生成端到端多模式X射线
EMIXER: End-to-end Multimodal X-ray Generation via Self-supervision
摘要：
深度生成模型使高质量数据的自动合成成为可能，以满足各种应用的需求。然而，最有效的生成模型专用于来自单个域的数据（例如，图像或文本）。诸如医疗保健的现实世界应用需要来自多个域的多模态数据（例如，图像和相应的文本），由于有限的可用性和隐私问题，这些图像和文本难以获取，并且更难以合成。为了解决这个联合合成的挑战，我们提出了一个端到端的多模态X射线生成模型（EMIXER），用于联合合成X射线图像和相应的自由文本报告，所有诊断标签的条件。EMIXER是一种条件生成对抗模型，通过1）基于标签生成图像，2）将图像编码为隐藏嵌入，3）通过分层解码器从图像嵌入中产生相应的文本，以及4）用于评估图像和相应文本的联合解码器。EMIXER还支持自我监督，以利用大量未标记的数据。使用真实X射线报告数据进行的大量实验说明了使用合成多模态样本的数据增强如何提高各种监督任务的性能，包括使用非常有限的样本进行COVID-19 X射线分类。生成的图像和报告的质量也由放射科医生确认。我们定量地表明，EMIXER生成的合成数据集可以增强X射线图像分类，报告生成模型，以实现仅在真实数据样本上训练的模型的5.94%和6.9%的改进。总之，我们的结果突出了生成模型的状态对推进临床机器学习的承诺。
英文摘要：
Deep generative models have enabled the automated synthesis of high-quality data for diverse applications. However, the most effective generative models are specialized to data from a single domain (e.g., images or text). Real-world applications such as healthcare require multi-modal data from multiple domains (e.g., both images and corresponding text), which are difficult to acquire due to limited availability and privacy concerns and are much harder to synthesize. To tackle this joint synthesis challenge, we propose an End-to-end MultImodal X-ray genERative model (EMIXER) for jointly synthesizing x-ray images and corresponding free-text reports, all conditional on diagnosis labels. EMIXER is an conditional generative adversarial model by 1) generating an image based on a label, 2) encoding the image to a hidden embedding, 3) producing the corresponding text via a hierarchical decoder from the image embedding, and 4) a joint discriminator for assessing both the image and the corresponding text. EMIXER also enables self-supervision to leverage vast amount of unlabeled data. Extensive experiments with real X-ray reports data illustrate how data augmentation using synthesized multimodal samples can improve the performance of a variety of supervised tasks including COVID-19 X-ray classification with very limited samples. The quality of generated images and reports are also confirmed by radiologists. We quantitatively show that EMIXER generated synthetic datasets can augment X-ray image classification, report generation models to achieve 5.94% and 6.9% improvement on models trained only on real data samples. Taken together, our results highlight the promise of state of generative models to advance clinical machine learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
