探索多模态LLM在医学影像中的组合泛化能力
Exploring Compositional Generalization of Multimodal LLMs for Medical Imaging
AI总结：
本文通过构建包含106个数据集的Med-MAT基准，验证组合泛化（CG）是多模态LLM在医学影像多任务训练中实现泛化的关键驱动力，并确认其在分类与检测任务及有限数据场景下的有效性。
摘要：
医学影像为诊断提供了重要的视觉见解，多模态大语言模型（MLLMs）因其强大的泛化能力被越来越多地用于其分析；然而，驱动这种泛化的潜在因素仍不清楚。当前研究表明，多任务训练优于单任务训练，因为不同任务可以相互受益，但它们往往忽略了这些任务内部的关联。为了分析这一现象，我们尝试采用组合泛化（CG）作为指导框架，CG指的是模型通过重组已学习的元素来理解新组合的能力。由于医学影像可以由模态、解剖区域和任务精确定义，自然为探索CG提供了环境，我们整合了106个医学数据集创建Med-MAT进行全面实验。实验证实MLLMs能利用CG理解未见医学影像，并确认CG是多任务训练中观察到的泛化的主要驱动因素之一。此外，进一步研究证明CG有效支持有限数据的数据集，并确认MLLMs能在分类和检测任务间实现CG，凸显其更广泛的泛化潜力。Med-MAT可在https://github.com/FreedomIntelligence/Med-MAT获取。
英文摘要：
Medical imaging provides essential visual insights for diagnosis, and multimodal large language models (MLLMs) are increasingly utilized for its analysis due to their strong generalization capabilities; however, the underlying factors driving this generalization remain unclear. Current research suggests that multi-task training outperforms single-task as different tasks can benefit each other, but they often overlook the internal relationships within these tasks. To analyze this phenomenon, we attempted to employ compositional generalization (CG), which refers to the models' ability to understand novel combinations by recombining learned elements, as a guiding framework. Since medical images can be precisely defined by Modality, Anatomical area, and Task, naturally providing an environment for exploring CG, we assembled 106 medical datasets to create Med-MAT for comprehensive experiments. The experiments confirmed that MLLMs can use CG to understand unseen medical images and identified CG as one of the main drivers of the generalization observed in multi-task training. Additionally, further studies demonstrated that CG effectively supports datasets with limited data and confirmed that MLLMs can achieve CG across classification and detection tasks, underscoring its broader generalization potential. Med-MAT is available at https://github.com/FreedomIntelligence/Med-MAT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
