arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.02176cs.CL
收藏

迈向参数高效:一种具有动态容量的分层稀疏激活 Transformer

Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity

Johns Hopkins University(约翰斯·霍普金斯大学);Meta AI

Haoran Xu, Maha Elbayad, Kenton Murray, Jean Maillard, Vedanuj Goswami

AI总结:

针对 MoE 模型因专家等容量导致的参数低效问题,提出分层稀疏激活的 SMoE 模型,通过分层结构为不同 token 动态分配容量,在三个多语言机器翻译基准上以相同或更少参数超越现有 MoE 模型。

摘要:

采用稀疏激活的混合专家模型在大幅增加参数数量的同时,能保持每个 token 较低的计算需求,已被证明有效。然而,近期研究表明 MoE 模型本质上存在参数低效问题,因为随着专家数量的增加,性能提升会逐渐递减。我们假设这种参数低效性是由于所有专家具有相同容量所致,这可能无法充分满足不同 token 或任务的复杂度需求。鉴于此,我们提出了分层混合专家模型,其具有分层结构,能为不同 token 分配动态容量。我们在三个多语言机器翻译基准上验证了 SMoE 的有效性,分别包含 4、15 和 94 个语言对。结果表明,SMoE 在参数量相同或更少的情况下,优于多种最先进的 MoE 模型。

英文摘要:

Mixture-of-experts (MoE) models that employ sparse activation have demonstrated effectiveness in significantly increasing the number of parameters while maintaining low computational requirements per token. However, recent studies have established that MoE models are inherently parameter-inefficient as the improvement in performance diminishes with an increasing number of experts. We hypothesize this parameter inefficiency is a result of all experts having equal capacity, which may not adequately meet the varying complexity requirements of different tokens or tasks. In light of this, we propose Stratified Mixture of Experts (SMoE) models, which feature a stratified structure and can assign dynamic capacity to different tokens. We demonstrate the effectiveness of SMoE on three multilingual machine translation benchmarks, containing 4, 15, and 94 language pairs, respectively. We show that SMoE outperforms multiple state-of-the-art MoE models with the same or fewer parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑