arXiv 2604.036352026-04-04cs.CVcs.AI

多模态病理生成模型基础框架

A Generative Foundation Model for Multimodal Histopathology

Department of Radiation Oncology, Stanford University School of Medicine（斯坦福大学医学院放射肿瘤学系）；Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine（斯坦福大学医学院精神病学与行为科学系）；Department of Statistics and Data Science, Yale University（耶鲁大学统计与数据科学系）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）；Department of Electrical Engineering, Stanford University（斯坦福大学电气工程系）；Department of Biomedical Data Science, Stanford University（斯坦福大学生物医学数据科学系）；Department of Radiology, Stanford University（斯坦福大学放射学系）；Center for Artificial Intelligence in Medicine and Imaging, Stanford University（斯坦福大学医学与影像人工智能中心）

Jinxi Xiang, Mingjie Li, Siyu Hou, Yijiang Chen, Xiangde Luo, Yuanfeng Ji, Xiang Zhou, Ehsan Adeli, Akshay Chaudhari, Curtis P. Langlotz, Kilian M. Pohl, Ruijiang Li

备注：: 33 pages, 9 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MuPD模型，通过扩散变压器整合病理图像、分子数据和临床文本，实现跨模态生成任务，提升诊断准确性和数据扩展性。

摘要：

准确诊断复杂疾病需要整合组织学、分子和临床数据，但实际中这些模态常不完整。现有方法依赖任务特定模型，限制了泛化能力。本文提出MuPD，通过扩散变压器整合H&E染色组织学、RNA表达谱和临床文本到共享潜在空间。预训练于1亿个组织学图像块、160万文本-组织学对和1080万RNA-组织学对，覆盖34种人体器官。MuPD支持多种跨模态合成任务，通过最小或无任务特定微调实现。对于文本条件和图像到图像生成，MuPD生成组织学忠实的组织架构，将FID分数降低50%并提升少样本分类准确率47%。对于RNA条件组织学生成，MuPD在五种癌症类型中保持细胞类型分布，FID降低23%。作为虚拟染色剂，MuPD将H&E图像转为免疫组化和多色免疫荧光，平均标记相关性提升37%。这些结果表明，跨异质病理模态预训练的单一生成模型可显著优于专用模型，提供可扩展的多模态病理计算框架。

英文摘要：

Accurate diagnosis and treatment of complex diseases require integrating histological, molecular, and clinical data, yet in practice these modalities are often incomplete owing to tissue scarcity, assay cost, and workflow constraints. Existing computational approaches attempt to impute missing modalities from available data but rely on task-specific models trained on narrow, single source-target pairs, limiting their generalizability. Here we introduce MuPD (Multimodal Pathology Diffusion), a generative foundation model that embeds hematoxylin and eosin (H&E)-stained histology, molecular RNA profiles, and clinical text into a shared latent space through a diffusion transformer with decoupled cross-modal attention. Pretrained on 100 million histology image patches, 1.6 million text-histology pairs, and 10.8 million RNA-histology pairs spanning 34 human organs, MuPD supports diverse cross-modal synthesis tasks with minimal or no task-specific fine-tuning. For text-conditioned and image-to-image generation, MuPD synthesizes histologically faithful tissue architectures, reducing Fréchet inception distance (FID) scores by 50% relative to domain-specific models and improving few-shot classification accuracy by up to 47% through synthetic data augmentation. For RNA-conditioned histology generation, MuPD reduces FID by 23% compared with the next-best method while preserving cell-type distributions across five cancer types. As a virtual stainer, MuPD translates H&E images to immunohistochemistry and multiplex immunofluorescence, improving average marker correlation by 37% over existing approaches. These results demonstrate that a single, unified generative model pretrained across heterogeneous pathology modalities can substantially outperform specialized alternatives, providing a scalable computational framework for multimodal histopathology.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。