MultiMed：大规模多模态与多任务的医学理解
MultiMed: Massively Multimodal and Multitask Medical Understanding
AI总结：
提出MultiMed基准，涵盖十种医学模态的256万样本与十一项任务，通过评估多模态多任务模型证明跨模态联合训练能提升医学预测性能与鲁棒性。
摘要：
生物医学数据本质上是多模态的，包括电子健康记录、医学影像、数字病理学、基因组测序、可穿戴传感器等。将人工智能工具应用于这些多方面的传感技术，有望彻底改变人类健康和疾病的预后、诊断和管理。然而，目前的生物医学AI方法通常仅使用一种或少数几种医学模态和任务进行训练和评估。这一局限性阻碍了能够利用许多异构生物医学传感器之间丰富互联信息的综合性工具的开发。为了应对这一挑战，我们提出了MultiMed，这是一个旨在评估并实现跨广泛医学模态和任务的大规模学习的基准。MultiMed包含跨越医学报告、病理学、基因组学和蛋白质数据等十种医学模态的256万个样本，并被组织成十一个具有挑战性的任务，包括疾病预后、蛋白质结构预测和医学问答。使用MultiMed，我们进行了全面的实验，对最先进的单模态、多模态和多任务模型进行了基准测试。我们的分析强调了在许多相关模态和任务中训练大规模医学模型的优势。此外，MultiMed支持研究跨相关医学概念的泛化能力、对真实世界噪声数据和分布偏移的鲁棒性，以及通过新颖的模态组合来提升预测性能。MultiMed将公开发布并定期更新，欢迎社区的反馈与贡献。
英文摘要：
Biomedical data is inherently multimodal, consisting of electronic health records, medical imaging, digital pathology, genome sequencing, wearable sensors, and more. The application of artificial intelligence tools to these multifaceted sensing technologies has the potential to revolutionize the prognosis, diagnosis, and management of human health and disease. However, current approaches to biomedical AI typically only train and evaluate with one or a small set of medical modalities and tasks. This limitation hampers the development of comprehensive tools that can leverage the rich interconnected information across many heterogeneous biomedical sensors. To address this challenge, we present MultiMed, a benchmark designed to evaluate and enable large-scale learning across a wide spectrum of medical modalities and tasks. MultiMed consists of 2.56 million samples across ten medical modalities such as medical reports, pathology, genomics, and protein data, and is structured into eleven challenging tasks, including disease prognosis, protein structure prediction, and medical question answering. Using MultiMed, we conduct comprehensive experiments benchmarking state-of-the-art unimodal, multimodal, and multitask models. Our analysis highlights the advantages of training large-scale medical models across many related modalities and tasks. Moreover, MultiMed enables studies of generalization across related medical concepts, robustness to real-world noisy data and distribution shifts, and novel modality combinations to improve prediction performance. MultiMed will be publicly available and regularly updated and welcomes inputs from the community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
