arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.04153cs.LGcs.AI
收藏

百万专家混合

Mixture of A Million Experts

Google DeepMind(谷歌DeepMind)

Xu Owen He

AI总结:

本文提出 PEER 层设计,利用乘积键技术从超过一百万个微小专家中稀疏检索,在语言建模任务上以更优的性能-计算权衡超越稠密 FFW 和粗粒度 MoE,为 Transformer 的高效扩展开辟新路径。

摘要:

标准 Transformer 架构中的前馈(FFW)层随着隐藏层宽度的增加,其计算成本和激活内存呈线性增长。稀疏专家混合(MoE)架构通过将模型规模与计算成本解耦,已成为解决这一问题的可行方法。最近发现的细粒度 MoE 缩放定律表明,更高的粒度能带来更好的性能。然而,由于计算和优化方面的挑战,现有 MoE 模型只能使用少量专家。本文提出了 PEER(parameter efficient expert retrieval,参数高效专家检索),这是一种新颖的层设计,利用乘积键技术从海量微小专家池(超过一百万个)中进行稀疏检索。在语言建模任务上的实验表明,PEER 层在性能与计算的权衡方面优于稠密 FFW 层和粗粒度 MoE。通过高效利用大量专家,PEER 在保持计算效率的同时,释放了进一步扩展 Transformer 模型的潜力。

英文摘要:

The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows. Sparse mixture-of-experts (MoE) architectures have emerged as a viable approach to address this issue by decoupling model size from computational cost. The recent discovery of the fine-grained MoE scaling law shows that higher granularity leads to better performance. However, existing MoE models are limited to a small number of experts due to computational and optimization challenges. This paper introduces PEER (parameter efficient expert retrieval), a novel layer design that utilizes the product key technique for sparse retrieval from a vast pool of tiny experts (over a million). Experiments on language modeling tasks demonstrate that PEER layers outperform dense FFWs and coarse-grained MoEs in terms of performance-compute trade-off. By enabling efficient utilization of a massive number of experts, PEER unlocks the potential for further scaling of transformer models while maintaining computational efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑