Med-MoE：面向轻量级医学视觉-语言模型的领域专用专家混合框架
Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models
AI总结：
本文提出轻量级医学多模态框架 Med-MoE，通过多模态对齐、指令微调与领域专用专家混合，在 Med-VQA 和图像分类任务上以仅约 30%-50% 的激活参数达到领先性能。
摘要：
近年来，通用或领域专用的多模态大语言模型（LLMs）在医学决策方面取得了显著进展。然而，这些模型通常被指定用于特定的分类或生成任务，并且需要在大规模数据集上进行模型训练或微调，涉及庞大的参数量和巨大的计算开销，这阻碍了它们在实际中多样化的资源受限场景下的临床实用性。本文提出了一种新颖且轻量级的框架 Med-MoE（专家混合），可同时处理判别式和生成式多模态医学任务。Med-MoE 的学习过程包括三个步骤：多模态医学对齐、指令微调与路由，以及领域专用 MoE 微调。在将多模态医学图像与 LLM token 对齐之后，我们通过指令微调使模型能够执行不同的多模态医学任务，并配备一个可训练的路由器，用于跨输入模态进行专家选择。最后，通过将路由器与多个领域专用专家集成来对模型进行微调，这些专家被选择性激活，并由元专家进一步增强。在 VQA-RAD、SLAKE 和 Path-VQA 等数据集上，针对开放式和封闭式医学视觉问答（Med-VQA）以及图像分类任务的综合实验表明，我们的模型能够取得优于或与最先进基线相当的性能，同时仅需激活约 30%-50% 的模型参数。广泛的分析和消融实验验证了该方法的有效性和实际实用性。
英文摘要：
Recent advancements in general-purpose or domain-specific multimodal large language models (LLMs) have witnessed remarkable progress for medical decision-making. However, they are designated for specific classification or generative tasks, and require model training or finetuning on large-scale datasets with sizeable parameters and tremendous computing, hindering their clinical utility across diverse resource-constrained scenarios in practice. In this paper, we propose a novel and lightweight framework Med-MoE (Mixture-of-Experts) that tackles both discriminative and generative multimodal medical tasks. The learning of Med-MoE consists of three steps: multimodal medical alignment, instruction tuning and routing, and domain-specific MoE tuning. After aligning multimodal medical images with LLM tokens, we then enable the model for different multimodal medical tasks with instruction tuning, together with a trainable router tailored for expert selection across input modalities. Finally, the model is tuned by integrating the router with multiple domain-specific experts, which are selectively activated and further empowered by meta expert. Comprehensive experiments on both open- and close-end medical question answering (Med-VQA) and image classification tasks across datasets such as VQA-RAD, SLAKE and Path-VQA demonstrate that our model can achieve performance superior to or on par with state-of-the-art baselines, while only requiring approximately 30\%-50\% of activated model parameters. Extensive analysis and ablations corroborate the effectiveness and practical utility of our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
