Large language models like ChatGPT have shown substantial progress in natural language understanding and generation, proving valuable across various disciplines, including the medical field. Despite advancements, challenges persist due to the complexity and diversity inherent in medical tasks which often require multi-task learning capabilities. Previous approaches, although beneficial, fall short in real-world applications because they necessitate task-specific annotations at inference time, limiting broader generalization. This paper introduces MING-MOE, a novel Mixture-of-Expert~(MOE)-based medical large language model designed to manage diverse and complex medical tasks without requiring task-specific annotations, thus enhancing its usability across extensive datasets. MING-MOE employs a Mixture of Low-Rank Adaptation (MoLoRA) technique, allowing for efficient parameter usage by maintaining base model parameters static while adapting through a minimal set of trainable parameters. We demonstrate that MING-MOE achieves state-of-the-art (SOTA) performance on over 20 medical tasks, illustrating a significant improvement over existing models. This approach not only extends the capabilities of medical language models but also improves inference efficiency.
EMTSF:Extraordinary Mixture of SOTA Models for Time Series Forecasting
Engineering, University of Bridgeport, Bridgeport, CT, USA(工程学院,布里奇波特大学);Department of Computer Science(计算机科学系);Statistics, University of Rhode Island, Kingston, RI, USA(统计学,罗德岛大学)
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);KTH Royal Institute of Technology(瑞典皇家理工学院);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
Guangzhou University(广州大学);Institue of Automation Chinese Academy of Sciences(中国科学院自动化研究所);ByteDance Inc.(字节跳动公司);University of Science and Technology Beijing(北京科技大学)
CuMA: Aligning LLMs with Sparse Cultural Values via Demographic-Aware Mixture of Adapters
Southeast University(东南大学);ByteDance Inc.(字节跳动公司);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用重点实验室(东南大学),中华人民共和国教育部,中国)
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
Inspur Genersoft Co. Ltd.(浪潮 Genersoft 有限公司);Inspur Group Co. Ltd.(浪潮集团有限公司);Interactive Entertainment Group, Tencent Inc(腾讯公司互动娱乐集团);Nanjing University of Information Science and Technology(南京信息工程大学);Northeastern University(东北大学)
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology(大连理工大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(香港科技大学)
RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
Tsinghua University(清华大学);Beijing Information Science and Technology University(北京信息科技大学);National University of Singapore(新加坡国立大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Beijing University of Posts and Telecommunications(北京邮电大学)
Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification
The University of Tokyo(东京大学);University of Southern California(南加州大学);The Chinese University of Hong Kong(香港中文大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心)