MedViLaM：一种具备先进泛化能力与可解释性的多模态大语言模型，用于医学数据理解与生成
MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation
AI总结：
本文提出统一视觉-语言模型 MedViLaM 及多任务基准 MultiMedBench，在多种医学任务上显著超越其他通才模型，并展现出零样本泛化与推理能力。
摘要：
医学本质上是多模态和多任务的，其多样化的数据模态涵盖文本和影像。然而，医学领域的大多数模型都是单模态单任务的，缺乏良好的泛化能力和可解释性。在本研究中，我们提出了 MedViLaM，一个统一的视觉-语言模型，旨在成为面向医学数据的通才模型，能够使用同一组模型权重灵活地编码和解释各种形式的医学数据，包括临床语言和影像。为了促进这种多任务模型的构建，我们构建了 MultiMedBench，一个综合性的预训练数据集和基准，包含若干不同的任务，即连续问答、多标签疾病分类、疾病定位、放射学报告生成与摘要。MedViLaM 在所有 MultiMedBench 任务上均展现出强大的性能，经常以显著优势超越其他通才模型。此外，我们展示了向新医学概念和任务的零样本泛化实例、跨不同任务的有效迁移学习，以及零样本医学推理能力的涌现。
英文摘要：
Medicine is inherently multimodal and multitask, with diverse data modalities spanning text, imaging. However, most models in medical field are unimodal single tasks and lack good generalizability and explainability. In this study, we introduce MedViLaM, a unified vision-language model towards a generalist model for medical data that can flexibly encode and interpret various forms of medical data, including clinical language and imaging, all using the same set of model weights. To facilitate the creation of such multi-task model, we have curated MultiMedBench, a comprehensive pretaining dataset and benchmark consisting of several distinct tasks, i.e., continuous question-answering, multi-label disease classification, disease localization, generation and summarization of radiology reports. MedViLaM demonstrates strong performance across all MultiMedBench tasks, frequently outpacing other generalist models by a significant margin. Additionally, we present instances of zero-shot generalization to new medical concepts and tasks, effective transfer learning across different tasks, and the emergence of zero-shot medical reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
