arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.02731cs.AI
收藏

面向通用任务指令微调的从稠密模型到混合专家模型的参数高效稀疏构建

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligent Laboratory(上海人工智能实验室);ChatEDA Tech(ChatEDA科技)

Haoyuan Wu, Haisheng Zheng, Zhuolun He, Bei Yu

AI总结:

针对LLM指令微调中模型容量受限问题,提出PESC方法,通过MoE架构将稠密模型转为稀疏模型并集成adapters,以最小参数增加扩展容量并降低计算开销,实验证明其最优稀疏模型超越GPT-3.5。

摘要:

大型语言模型(LLMs)在通用自然语言处理(NLP)任务中展现出相当的能力。指令微调作为一种成功的范式,增强了LLMs遵循自然语言指令的能力,并在通用任务中表现出稳健的泛化能力。然而,由于模型容量受限,这些模型在多个任务上常遇到性能瓶颈。在指令微调阶段扩展此容量面临重大挑战。为解决这一问题,我们引入了参数高效稀疏构建(PESC),该方法利用混合专家架构将稠密模型构建为稀疏模型。PESC将adapters集成到稀疏模型的MoE层中,在不改变这些层内个体权重的情况下区分专家。与原始稀疏上采样相比,该方法在保证函数空间近似质量的前提下,通过最小化参数增加来促进模型容量扩展,显著降低了计算成本和GPU内存需求。我们的实证评估证明了PESC方法的有效性。在指令微调中使用PESC,我们最佳的稀疏模型优于其他稀疏和稠密模型,并展现出比GPT-3.5更优越的通用能力。代码可在https://github.com/wuhy68/Parameter-Efficient-MoE获取。

英文摘要:

Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing (NLP) tasks. Instruction tuning, a successful paradigm, enhances the ability of LLMs to follow natural language instructions and exhibit robust generalization across general tasks. However, these models often encounter performance limitations across multiple tasks due to constrained model capacity. Expanding this capacity during the instruction tuning phase poses significant challenges. To address this issue, we introduce parameter-efficient sparsity crafting (PESC), which crafts dense models into sparse models using the mixture-of-experts (MoE) architecture. PESC integrates adapters into the MoE layers of sparse models, differentiating experts without altering the individual weights within these layers. This method significantly reduces computational costs and GPU memory requirements, facilitating model capacity expansion through a minimal parameter increase when guaranteeing the quality of approximation in function space compared to original sparse upcycling. Our empirical evaluation demonstrates the effectiveness of the PESC method. Using PESC during instruction tuning, our best sparse model outperforms other sparse and dense models and exhibits superior general capabilities compared to GPT-3.5. Our code is available at https://github.com/wuhy68/Parameter-Efficient-MoE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑