arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.18339cs.CLcs.AI
收藏

当MOE遇见LLMs:面向多任务医疗应用的参数高效微调

When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications

City University of Hong Kong(香港城市大学);Jarvis Research Center(贾维斯研究中心);Tencent YouTu Lab(腾讯优图实验室);Southern University of Science and Technology(南方科技大学);University of Science and Technology of China(中国科学技术大学)

Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, Yefeng Zheng

AI总结:

针对医疗应用LLM微调中的任务多样性和参数消耗问题,提出参数高效微调框架MOELoRA,结合混合专家与低秩适配,通过任务驱动门控生成任务专属参数,在多任务医疗数据集上优于现有方法。

摘要:

大型语言模型(LLMs)最近的激增在众多领域引起了广泛关注。为了使通用LLMs适应特定领域(例如基于Web的医疗保健系统),通常需要进行微调。然而,在为医疗应用微调LLMs时会出现两个问题。一个是任务多样性问题,涉及真实医疗场景中的不同任务。这种多样性通常会导致因数据不平衡和跷跷板问题而产生的次优微调。此外,LLMs中的大量参数导致微调消耗巨大的时间和计算资源。为了解决这两个问题,我们提出了一种面向多任务医疗应用的新型参数高效微调框架,称为MOELoRA。该设计的框架旨在吸收混合专家(MOE)用于多任务学习和低秩适配(LoRA)用于参数高效微调的双重优势。为了统一MOE和LoRA,我们将多个专家设计为可训练参数,其中每个专家由一对低秩矩阵组成,以保持可训练参数的小规模。然后,为所有MOELoRA层提出了一种任务驱动的门控函数,该函数可以控制每个专家的贡献,并为各种任务生成不同的参数。我们在多任务医疗数据集上进行了实验,表明MOELoRA优于现有的参数高效微调方法。代码已在线提供。

英文摘要:

The recent surge in Large Language Models (LLMs) has garnered significant attention across numerous fields. Fine-tuning is often required to fit general LLMs for a specific domain, like the web-based healthcare system. However, two problems arise during fine-tuning LLMs for medical applications. One is the task variety problem, which involves distinct tasks in real-world medical scenarios. The variety often leads to sub-optimal fine-tuning for data imbalance and seesaw problems. Besides, the large amount of parameters in LLMs leads to huge time and computation consumption by fine-tuning. To address these two problems, we propose a novel parameter efficient fine-tuning framework for multi-task medical applications, dubbed as MOELoRA. The designed framework aims to absorb both the benefits of mixture-of-expert (MOE) for multi-task learning and low-rank adaptation (LoRA) for parameter efficient fine-tuning. For unifying MOE and LoRA, we devise multiple experts as the trainable parameters, where each expert consists of a pair of low-rank matrices to retain the small size of trainable parameters. Then, a task-motivated gate function for all MOELoRA layers is proposed, which can control the contributions of each expert and produce distinct parameters for various tasks. We conduct experiments on a multi-task medical dataset, indicating MOELoRA outperforms the existing parameter efficient fine-tuning methods. The code is available online.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑