arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.07035cs.LGcs.AI
收藏

HyperRouter:迈向稀疏专家混合模型的高效训练与推理

HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts

University of Tennessee at Chattanooga(田纳西大学查塔努加分校);VinUniversity;Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK(格勒诺布尔阿尔卑斯大学、法国国家信息与自动化研究所、法国国家科学研究中心、格勒诺布尔理工学院、LJK实验室);University of Science, Vietnam National University Ho Chi Minh City(胡志明市国家大学科学大学);Salesforce Research Asia(salesforce 亚洲研究院);Singapore Management University(新加坡管理大学)

Giang Do, Khiem Le, Quang Pham, TrungTin Nguyen, Thanh-Nam Doan, Bint T. Nguyen, Chenghao Liu, Savitha Ramasamy, Xiaoli Li, Steven Hoi

AI总结:

针对稀疏专家混合模型固定路由策略的次优性与效率不足问题,提出HyperRouter,通过固定超网络和可训练嵌入动态生成路由参数,平衡路由训练与冻结,在多任务上实现了更优性能与效率。

摘要:

稀疏专家混合模型(Sparse Mixture-of-Experts)通过将输入令牌仅路由到少数拆分专家,实现了大语言模型的高效训练。近期研究发现,固定路由函数可通过缓解坍缩问题(即所有专家最终学习到相似表示)取得具有竞争力的性能,但该策略存在两个关键局限:(i)随机路由函数得到的策略可能并非最优;(ii)其训练和评估阶段需要大量资源,导致效率提升有限。本研究提出HyperRouter,通过固定超网络和可训练嵌入动态生成路由函数的参数,在训练路由函数与冻结路由函数之间取得平衡,以学习更优的路由策略。在广泛任务上开展的大量实验表明,与现有路由方法相比,HyperRouter具备更优的性能与效率提升。我们的实现已公开于https://github.com/giangdip2410/HyperRouter。

英文摘要:

By routing input tokens to only a few split experts, Sparse Mixture-of-Experts has enabled efficient training of large language models. Recent findings suggest that fixing the routers can achieve competitive performance by alleviating the collapsing problem, where all experts eventually learn similar representations. However, this strategy has two key limitations: (i) the policy derived from random routers might be sub-optimal, and (ii) it requires extensive resources during training and evaluation, leading to limited efficiency gains. This work introduces \HyperRout, which dynamically generates the router's parameters through a fixed hypernetwork and trainable embeddings to achieve a balance between training the routers and freezing them to learn an improved routing policy. Extensive experiments across a wide range of tasks demonstrate the superior performance and efficiency gains of \HyperRouter compared to existing routing methods. Our implementation is publicly available at {\url{https://github.com/giangdip2410/HyperRouter}}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑