AT-MoE：基于LoRA方法的自适应任务规划混合专家模型
AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approach
AI总结：
本文提出AT-MoE架构，通过LoRA训练任务特定专家并结合逐层自适应分组路由模块，在保持可解释性和可控性的同时实现复杂任务下的最优专家融合，解决现有MoE在专业领域任务特定学习不足的问题。
摘要：
大型语言模型（LLMs）的出现开启了人工智能的新纪元，有望通过自动化和深入分析变革多个领域。混合专家（MoE）架构已被提出作为提升复杂任务中模型性能的解决方案。然而，现有的MoE模型在任务特定学习和可解释性方面存在不足，尤其是在医学等对精确性要求极高的领域。本文介绍了自适应任务规划混合专家模型（AT-MoE），这是一种旨在解决上述局限性的创新架构。我们首先通过LoRA方法训练任务特定专家，以增强专业领域的问题解决能力和可解释性。随后，我们引入了一个逐层自适应分组路由模块，该模块根据复杂任务指令优化模块融合，确保最优的任务解决效果。分组路由模块首先从专家组维度进行整体权重分配，然后在组内进行局部权重归一化调整。这种设计保持了多维度的平衡性、可控性和可解释性，同时促进了响应复杂指令的任务特定融合。
英文摘要：
The advent of Large Language Models (LLMs) has ushered in a new era of artificial intelligence, with the potential to transform various sectors through automation and insightful analysis. The Mixture of Experts (MoE) architecture has been proposed as a solution to enhance model performance in complex tasks. Yet, existing MoE models struggle with task-specific learning and interpretability, especially in fields like medicine where precision is critical. This paper introduces the Adaptive Task-planing Mixture of Experts(AT-MoE), an innovative architecture designed to address these limitations. We first train task-specific experts via LoRA approach to enhance problem-solving capabilities and interpretability in specialized areas. Subsequently, we introduce a layer-wise adaptive grouped routing module that optimizes module fusion based on complex task instructions, ensuring optimal task resolution. The grouped routing module first perform overall weight allocation from the dimension of the expert group, and then conduct local weight normalization adjustments within the group. This design maintains multi-dimensional balance, controllability, and interpretability, while facilitating task-specific fusion in response to complex instructions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
