arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.09027cs.CL
收藏

MING-MOE:通过稀疏低秩适配器专家混合增强大语言模型的医学多任务学习

MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Experts

Cooperative Medianet Innovation Center(合作 Medianet 创新中心);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)

Yusheng Liao, Shuyang Jiang, Yu Wang, Yanfeng Wang

AI总结:

针对医学大模型多任务泛化受限、推理需任务标注的问题,本文提出基于MoLoRA技术的MING-MOE模型,在20余项医学任务上达SOTA,兼顾性能与推理效率。

摘要:

ChatGPT等大语言模型在自然语言理解与生成方面已取得显著进展,在包括医学领域在内的多个学科中展现出应用价值。尽管已有进步,但由于医学任务固有的复杂性与多样性(通常需要多任务学习能力),相关挑战依然存在。此前的方法虽有裨益,但在实际应用中存在不足,因为它们在推理时需要任务特定的标注,限制了更广泛的泛化能力。本文提出MING-MOE,这是一种新型的基于专家混合(MOE)的医学大语言模型,旨在无需任务特定标注的情况下处理多样且复杂的医学任务,从而提升其在大规模数据集上的可用性。MING-MOE采用低秩适配混合(MoLoRA)技术,通过保持基础模型参数静态、仅通过少量可训练参数进行适配,实现高效的参数利用。我们证明,MING-MOE在20余项医学任务上达到了当前最优(SOTA)性能,相比现有模型有显著提升。该方法不仅拓展了医学语言模型的能力,还提高了推理效率。

英文摘要:

Large language models like ChatGPT have shown substantial progress in natural language understanding and generation, proving valuable across various disciplines, including the medical field. Despite advancements, challenges persist due to the complexity and diversity inherent in medical tasks which often require multi-task learning capabilities. Previous approaches, although beneficial, fall short in real-world applications because they necessitate task-specific annotations at inference time, limiting broader generalization. This paper introduces MING-MOE, a novel Mixture-of-Expert~(MOE)-based medical large language model designed to manage diverse and complex medical tasks without requiring task-specific annotations, thus enhancing its usability across extensive datasets. MING-MOE employs a Mixture of Low-Rank Adaptation (MoLoRA) technique, allowing for efficient parameter usage by maintaining base model parameters static while adapting through a minimal set of trainable parameters. We demonstrate that MING-MOE achieves state-of-the-art (SOTA) performance on over 20 medical tasks, illustrating a significant improvement over existing models. This approach not only extends the capabilities of medical language models but also improves inference efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑