arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.04339cs.LGcs.CL
收藏

在任务参数子空间中通过匹配模型进行合并

Merging by Matching Models in Task Parameter Subspaces

University of Toronto(多伦多大学);Vector Institute(向量研究所);University of North Carolina(北卡罗来纳大学)

Derek Tam, Mohit Bansal, Colin Raffel

AI总结:

本研究提出MaTS模型合并框架,将模型合并转化为任务参数子空间下的线性方程组求解问题,采用共轭梯度法突破闭式解局限,在多任务和中间任务合并上取得SOTA结果并开源相关资源。

摘要:

模型合并旨在以低成本将多个独立的特定任务模型组合为单个多任务模型。在本研究中,我们将以往的合并方法视为利用不同的“任务参数子空间”概念——模型在合并前会在该子空间中进行匹配。我们将给定模型的任务参数子空间与其损失景观相关联,并形式化阐述了这种模型合并方法可被视为求解一个线性方程组。以往的研究通常局限于存在闭式解的线性系统,而我们考虑使用共轭梯度法来求解。我们表明,使用共轭梯度法的效果优于闭式解,能够通过原本难以求解的线性系统实现合并,并且可以灵活选择多种初始化方式和“任务参数子空间”的估计方法。我们最终证明,名为“在任务参数子空间中匹配模型”(Matching Models in their Task Parameter Subspace, MaTS)的合并框架在多任务和中间任务模型合并任务上取得了当前最优结果。我们已将本研究使用的所有代码和检查点发布在https://github.com/r-three/mats。

英文摘要:

Model merging aims to cheaply combine individual task-specific models into a single multitask model. In this work, we view past merging methods as leveraging different notions of a ''task parameter subspace'' in which models are matched before being merged. We connect the task parameter subspace of a given model to its loss landscape and formalize how this approach to model merging can be seen as solving a linear system of equations. While past work has generally been limited to linear systems that have a closed-form solution, we consider using the conjugate gradient method to find a solution. We show that using the conjugate gradient method can outperform closed-form solutions, enables merging via linear systems that are otherwise intractable to solve, and flexibly allows choosing from a wide variety of initializations and estimates for the ''task parameter subspace''. We ultimately demonstrate that our merging framework called ''Matching Models in their Task Parameter Subspace'' (MaTS) achieves state-of-the-art results in multitask and intermediate-task model merging. We release all of the code and checkpoints used in our work at https://github.com/r-three/mats.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑