When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications
City University of Hong Kong(香港城市大学);Jarvis Research Center(贾维斯研究中心);Tencent YouTu Lab(腾讯优图实验室);Southern University of Science and Technology(南方科技大学);University of Science and Technology of China(中国科学技术大学)
The recent surge in Large Language Models (LLMs) has garnered significant attention across numerous fields. Fine-tuning is often required to fit general LLMs for a specific domain, like the web-based healthcare system. However, two problems arise during fine-tuning LLMs for medical applications. One is the task variety problem, which involves distinct tasks in real-world medical scenarios. The variety often leads to sub-optimal fine-tuning for data imbalance and seesaw problems. Besides, the large amount of parameters in LLMs leads to huge time and computation consumption by fine-tuning. To address these two problems, we propose a novel parameter efficient fine-tuning framework for multi-task medical applications, dubbed as MOELoRA. The designed framework aims to absorb both the benefits of mixture-of-expert (MOE) for multi-task learning and low-rank adaptation (LoRA) for parameter efficient fine-tuning. For unifying MOE and LoRA, we devise multiple experts as the trainable parameters, where each expert consists of a pair of low-rank matrices to retain the small size of trainable parameters. Then, a task-motivated gate function for all MOELoRA layers is proposed, which can control the contributions of each expert and produce distinct parameters for various tasks. We conduct experiments on a multi-task medical dataset, indicating MOELoRA outperforms the existing parameter efficient fine-tuning methods. The code is available online.
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);KTH Royal Institute of Technology(瑞典皇家理工学院);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
DeepFusion: Accelerating MoE Training via Federated Knowledge Distillation from Heterogeneous Edge Devices
School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦女王玛丽大学电子工程与计算机科学学院);Department of Computer Science, Faculty of Environment, Science and Economy, University of Exeter(埃克塞特大学环境、科学与经济学院计算机科学系);School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院);Beijing Key Laboratory of Petroleum Data Mining, China University of Petroleum(北京石油数据挖掘重点实验室,中国石油大学)
Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology(香港科技大学电子及计算机工程学系);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程学系);Institute of Medical Intelligence and XR, The Chinese University of Hong Kong(香港中文大学医学智能与扩展现实研究所);Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人创新中心)
SparseDoctor: Towards Efficient Chat Doctor with Mixture of Experts Enhanced Large Language Models
Department of Computer Science, Hong Kong Chu Hai College(香港 Chu Hai 学院计算机科学系);Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系)
College of Cyber Security, Jinan University(暨南大学网络空间安全学院);School of Information Engineering, Guangdong Eco-Engineering Polytechnic(广东生态工程职业学院信息工程学院);Shenzhen People’s Hospital (The Second Clinical Medical College, Jinan University)(深圳市人民医院(暨南大学第二临床医学院));University of Illinois Chicago(伊利诺伊大学芝加哥分校)
Conditional Rank Allocation for Taxonomy-Aware Medical Language Model Adaptation
National University of Singapore(新加坡国立大学);Shanghai General Hospital(上海市第一人民医院);Shanghai Jiao Tong University School of Medicine(上海交通大学医学院);University of Pennsylvania(宾夕法尼亚大学);Shandong University(山东大学);Syracuse University(雪城大学)
Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMs
Department of Biomedical Systems Informatics, Yonsei University College of Medicine(生物医学系统信息学系,延世大学医学院);Yonsei University College of Medicine(延世大学医学院);Department of Radiation Oncology, Yonsei University College of Medicine(放射肿瘤学系,延世大学医学院);Department of Biomedical Systems Informatics and the Department of Psychiatry, Yonsei University College of Medicine(生物医学系统信息学系和精神病学系,延世大学医学院);Institute for Innovation in Digital Healthcare, Yonsei University(数字医疗创新研究所,延世大学)
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
Google(谷歌);University of California, Berkeley(加州大学伯克利分校);Massachusetts Institute of Technology(麻省理工学院);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);The University of Texas at Austin(得克萨斯大学奥斯汀分校)
Flexible and Adaptable Summarization via Expertise Separation
KAUST(阿卜杜拉国王科技大学);MBZUAI(Mohamed bin Zayed 人工智能大学);Ant Group(蚂蚁集团);Shandong University(山东大学);Peking University(北京大学);Renmin University of China(中国人民大学);University of Notre Dame(圣母大学)
Correctness Coverage Evaluation for Medical Multiple-Choice Question Answering Based on the Enhanced Conformal Prediction Framework
School of Electronic and Information Engineering, Tongji University(同济大学电子与信息工程学院);School of Electrical and Information Engineering, Hunan University(湖南大学电气与信息工程学院);School of Economics and Management, Fuzhou University(福州市经济学与管理学院)
Language Models Meet Anomaly Detection for Better Interpretability and Generalizability
Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Klinikum Rechts der Isar(Rechts der Isar 医院);Imperial College London(伦敦帝国学院);Helmholtz AI(亥姆霍兹人工智能研究所);Helmholtz Munich(亥姆霍兹慕尼黑中心);King’s College London(伦敦国王学院)