arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.08274cs.LG
收藏

BAM! 就这么简单:面向混合专家模型的简单高效参数升级利用方法

BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts

Cohere For AI;Cohere;University of Oxford(牛津大学)

Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar, Simon Guo, David Cairuz, Bharat Venkitesh, Jakob Foerster, Phil Blunsom, Sebastian Ruder, Ahmet Ustun, Acyr Locatelli

AI总结:

针对稠密模型转MoE时仅能复用FFN层参数的局限,提出BAM方法,通过复用注意力参数构建MoA层、采用并行注意力架构提升效率,在相同计算数据约束下性能优于基线。

摘要:

混合专家(Mixture of Experts, MoE)框架凭借相较于稠密模型更优异的性能,已成为大语言模型的主流架构。然而,在大规模场景下从头训练MoE的成本高得令人望而却步。现有方法通过独立预训练多个稠密专家模型、再用其初始化MoE的方式缓解这一问题,具体做法是用专家的前馈网络(feed-forward network, FFN)初始化MoE的专家层,同时合并其他参数。但该方法仅能复用稠密模型的FFN层参数,限制了将这类模型“升级改造”为MoE时的性能增益。\n我们提出BAM(Branch-Attend-Mix,分支-注意力-混合)这一简单且有效的方法来解决上述缺陷。BAM充分利用专用稠密模型的参数:不仅用其FFN初始化MoE层,还将专家的注意力参数完整复用以初始化混合注意力(Mixture of Attention, MoA)层的软变体。我们探索了两种升级利用注意力参数的方案:1)从稠密模型初始化独立的注意力专家,保留全部注意力参数以实现最优模型性能;2)在所有专家间共享键(key)和值(value)参数,以提升推理效率。为进一步提高效率,我们为MoE采用并行注意力Transformer架构,使注意力专家和FFN专家可并行计算。\n我们在参数量从5.9亿到20亿不等的种子模型上开展实验,结果表明,在相同的计算和数据约束下,BAM在困惑度和下游任务表现上均优于基线方法。

英文摘要:

The Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance over dense models. However, training MoEs from scratch in a large-scale regime is prohibitively expensive. Existing methods mitigate this by pre-training multiple dense expert models independently and using them to initialize an MoE. This is done by using experts' feed-forward network (FFN) to initialize the MoE's experts while merging other parameters. However, this method limits the reuse of dense model parameters to only the FFN layers, thereby constraining the advantages when "upcycling" these models into MoEs. We propose BAM (Branch-Attend-Mix), a simple yet effective method that addresses this shortcoming. BAM makes full use of specialized dense models by not only using their FFN to initialize the MoE layers but also leveraging experts' attention parameters fully by initializing them into a soft-variant of Mixture of Attention (MoA) layers. We explore two methods for upcycling attention parameters: 1) initializing separate attention experts from dense models including all attention parameters for the best model performance; and 2) sharing key and value parameters across all experts to facilitate for better inference efficiency. To further improve efficiency, we adopt a parallel attention transformer architecture to MoEs, which allows the attention experts and FFN experts to be computed concurrently. Our experiments on seed models ranging from 590 million to 2 billion parameters demonstrate that BAM surpasses baselines in both perplexity and downstream task performance, within the same computational and data constraints.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑