arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.09770cs.LGcs.AI
收藏

基于混合专家模型融合的高效帕累托集近似方法研究

Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion

School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);University of Oxford(牛津大学);School of Computer Science, Beijing Institute of Technology(北京理工大学计算机学院);Nanyang Technological University(南洋理工大学)

Anke Tang, Li Shen, Yong Luo, Shiwei Liu, Han Hu, Bo Du, Dacheng Tao

AI总结:

本文提出基于混合专家模型融合的方法,解决大型深度神经网络多目标优化的帕累托集近似问题,该方法高效且可扩展,内存占用低,在视觉与语言任务上表现优异。

摘要:

针对大型深度神经网络的多目标优化问题,由于损失景观的复杂性以及模型训练与评估的计算成本高昂,该任务极具挑战性。对大型模型进行高效帕累托前沿近似,可支撑多任务学习、权衡分析等各类任务的多目标优化。现有帕累托集学习算法存在两类问题:一是进化算法、超网络算法、超体积最大化方法等计算成本高昂,对大型模型的可扩展性受限;二是标量化算法需为每个目标射线单独训练模型,学习完整帕累托集的效率低下,且无法有效捕捉目标间的权衡关系。受近期模型合并技术成功的启发,本文提出一种实用且可扩展的帕累托集学习方法,该方法基于混合专家(Mixture of Experts,MoE)的模型融合技术,通过集成专用单任务模型的权重,MoE模块可有效捕捉多目标间的权衡关系,紧密近似大型神经网络的完整帕累托集。一旦路由器完成学习并设置偏好向量,即可卸载MoE模块,推理阶段不会引入额外计算成本。本文在CLIP-ViT、GPT-2等大型模型的视觉与语言任务上开展大量实验,结果表明,所提方法可高效近似大型模型的完整帕累托前沿;仅使用MoE路由器的数百个可训练参数,其内存使用量甚至低于线性标量化算法与学习单个帕累托最优解的算法,且对目标数量与模型规模均具有可扩展性。

英文摘要:

Solving multi-objective optimization problems for large deep neural networks is a challenging task due to the complexity of the loss landscape and the expensive computational cost of training and evaluating models. Efficient Pareto front approximation of large models enables multi-objective optimization for various tasks such as multi-task learning and trade-off analysis. Existing algorithms for learning Pareto set, including (1) evolutionary, hypernetworks, and hypervolume-maximization methods, are computationally expensive and have restricted scalability to large models; (2) Scalarization algorithms, where a separate model is trained for each objective ray, which is inefficient for learning the entire Pareto set and fails to capture the objective trade-offs effectively. Inspired by the recent success of model merging, we propose a practical and scalable approach to Pareto set learning problem via mixture of experts (MoE) based model fusion. By ensembling the weights of specialized single-task models, the MoE module can effectively capture the trade-offs between multiple objectives and closely approximate the entire Pareto set of large neural networks. Once the routers are learned and a preference vector is set, the MoE module can be unloaded, thus no additional computational cost is introduced during inference. We conduct extensive experiments on vision and language tasks using large-scale models such as CLIP-ViT and GPT-2. The experimental results demonstrate that our method efficiently approximates the entire Pareto front of large models. Using only hundreds of trainable parameters of the MoE routers, our method even has lower memory usage compared to linear scalarization and algorithms that learn a single Pareto optimal solution, and are scalable to both the number of objectives and the size of the model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑