Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);University of Oxford(牛津大学);School of Computer Science, Beijing Institute of Technology(北京理工大学计算机学院);Nanyang Technological University(南洋理工大学)
Anke Tang, Li Shen, Yong Luo, Shiwei Liu, Han Hu, Bo Du, Dacheng Tao
针对大型深度神经网络的多目标优化问题,由于损失景观的复杂性以及模型训练与评估的计算成本高昂,该任务极具挑战性。对大型模型进行高效帕累托前沿近似,可支撑多任务学习、权衡分析等各类任务的多目标优化。现有帕累托集学习算法存在两类问题:一是进化算法、超网络算法、超体积最大化方法等计算成本高昂,对大型模型的可扩展性受限;二是标量化算法需为每个目标射线单独训练模型,学习完整帕累托集的效率低下,且无法有效捕捉目标间的权衡关系。受近期模型合并技术成功的启发,本文提出一种实用且可扩展的帕累托集学习方法,该方法基于混合专家(Mixture of Experts,MoE)的模型融合技术,通过集成专用单任务模型的权重,MoE模块可有效捕捉多目标间的权衡关系,紧密近似大型神经网络的完整帕累托集。一旦路由器完成学习并设置偏好向量,即可卸载MoE模块,推理阶段不会引入额外计算成本。本文在CLIP-ViT、GPT-2等大型模型的视觉与语言任务上开展大量实验,结果表明,所提方法可高效近似大型模型的完整帕累托前沿;仅使用MoE路由器的数百个可训练参数,其内存使用量甚至低于线性标量化算法与学习单个帕累托最优解的算法,且对目标数量与模型规模均具有可扩展性。
英文摘要:
Solving multi-objective optimization problems for large deep neural networks is a challenging task due to the complexity of the loss landscape and the expensive computational cost of training and evaluating models. Efficient Pareto front approximation of large models enables multi-objective optimization for various tasks such as multi-task learning and trade-off analysis. Existing algorithms for learning Pareto set, including (1) evolutionary, hypernetworks, and hypervolume-maximization methods, are computationally expensive and have restricted scalability to large models; (2) Scalarization algorithms, where a separate model is trained for each objective ray, which is inefficient for learning the entire Pareto set and fails to capture the objective trade-offs effectively. Inspired by the recent success of model merging, we propose a practical and scalable approach to Pareto set learning problem via mixture of experts (MoE) based model fusion. By ensembling the weights of specialized single-task models, the MoE module can effectively capture the trade-offs between multiple objectives and closely approximate the entire Pareto set of large neural networks. Once the routers are learned and a preference vector is set, the MoE module can be unloaded, thus no additional computational cost is introduced during inference. We conduct extensive experiments on vision and language tasks using large-scale models such as CLIP-ViT and GPT-2. The experimental results demonstrate that our method efficiently approximates the entire Pareto front of large models. Using only hundreds of trainable parameters of the MoE routers, our method even has lower memory usage compared to linear scalarization and algorithms that learn a single Pareto optimal solution, and are scalable to both the number of objectives and the size of the model.
MC#: Mixture Compressor for Mixture-of-Experts Large Models
Department of Electrical and Electronic Engineering, The University of Hong Kong(香港大学电子与电气工程系);School of Computing, National University of Singapore(新加坡国立大学计算机学院);NVIDIA Research(NVIDIA研究)
HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts
Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学);INF Technology(INF科技);Hong Kong University of Science and Technology(香港科技大学)
The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models
The Fin AI(Fin AI);Georgia Institute of Technology(佐治亚理工学院);Cornell University(康奈尔大学);Stevens Institute of Technology(史蒂文斯理工学院);The University of Manchester(曼彻斯特大学)
DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Nanyang Technological University(南洋理工大学);University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学)
Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
Department of Computer Science and Engineering, Korea University, Republic of Korea(韩国大学计算机科学与工程系);KT, Republic of Korea(韩国KT);School of Software, Soongsil University, Republic of Korea(顺天大学软件学院)
Booth School of Business University of Chicago(芝加哥大学商学院);Volgenau School of Engineering, George Mason University(乔治·马歇尔大学工程学院);Department of Systems Engineering and Operations Research George Mason University(乔治·马歇尔大学系统工程与运营管理系)