MAP：基于二次近似的摊销帕累托前沿的低计算量模型合并
MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation
AI总结：
本文提出低计算量的模型合并算法MAP，利用二次近似摊销帕累托前沿评估成本，在视觉和NLP任务上准确识别权衡解，并扩展贝叶斯MAP和嵌套MAP以适应不同任务数量。
摘要：
模型合并已成为将多个单任务模型组合成多任务模型的有效方法。该过程通常涉及计算模型参数的加权平均值，而无需任何额外训练。现有的模型合并方法侧重于提高平均任务精度。然而，不同任务目标之间的干扰和冲突可能导致合并过程中的权衡。在现实应用中，一组具有不同权衡的解决方案可能更具信息性，帮助从业者根据多样化的偏好做出决策。在本文中，我们引入了一种新颖且低计算量的算法，即摊销帕累托前沿模型合并（MAP）。MAP高效地识别用于合并多个模型的缩放系数帕累托集，反映所涉及的权衡。它通过使用从预选缩放系数集导出的二次近似代理模型，摊销了估计帕累托前沿所需的大量评估计算成本。在视觉和自然语言处理任务上的实验结果表明，MAP能够准确识别帕累托前沿，为从业者提供灵活的解决方案以平衡竞争性任务目标。我们还引入了贝叶斯MAP，用于任务数量相对较少的场景，以及嵌套MAP，用于任务数量较多的情况，进一步降低评估的计算成本。
英文摘要：
Model merging has emerged as an effective approach to combine multiple single-task models into a multitask model. This process typically involves computing a weighted average of the model parameters without any additional training. Existing model-merging methods focus on enhancing average task accuracy. However, interference and conflicts between the objectives of different tasks can lead to trade-offs during the merging process. In real-world applications, a set of solutions with various trade-offs can be more informative, helping practitioners make decisions based on diverse preferences. In this paper, we introduce a novel and low-compute algorithm, Model Merging with Amortized Pareto Front (MAP). MAP efficiently identifies a Pareto set of scaling coefficients for merging multiple models, reflecting the trade-offs involved. It amortizes the substantial computational cost of evaluations needed to estimate the Pareto front by using quadratic approximation surrogate models derived from a pre-selected set of scaling coefficients. Experimental results on vision and natural language processing tasks demonstrate that MAP can accurately identify the Pareto front, providing practitioners with flexible solutions to balance competing task objectives. We also introduce Bayesian MAP for scenarios with a relatively low number of tasks and Nested MAP for situations with a high number of tasks, further reducing the computational cost of evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
