arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.26693cs.LGcs.AIstat.ML
收藏

损失景观上的模型合并:几何视角

Model Merging on Loss Landscape: A Geometry Perspective

Purdue University(普渡大学);Waymo LLC(Waymo公司)

Juanwu Lu, Anand Bhaskar, Brian Axelrod, Ekaterina Tolstaya, Tristan Emrich

AI总结:

提出EpiMer框架,将模型合并视为黎曼流形上的Fréchet均值,利用任务向量张成的低秩子空间和期望Hessian度量,理论证明曲率感知合并优于平坦几何方法,并在八个图像分类任务上验证了性能提升。

摘要:

模型合并为无需重新训练的知识集成和并行开发提供了有前景的途径。然而,现有方法要么忽略损失景观的几何结构,要么依赖于难以处理的全空间Hessian近似。我们提出EpiMer,一个将模型合并视为黎曼流形上Fréchet均值求解的框架,并将计算限制在由任务向量张成的低秩子空间内。以期望Hessian作为度量,我们揭示了局部曲率与参数认知不确定性之间的联系。我们的理论分析将合并误差界分解为子空间Fréchet方差和残差能量,并提供了曲率感知合并何时在理论上优于平坦几何方法的闭式刻画。此外,我们的框架将曲率感知方法和最近的谱方法统一为不同几何度量下子空间Fréchet均值的特例。在八个图像分类任务上合并微调的CLIP-ViT模型,Epistemic Merging在匹配秩下严格优于所有三个CLIP-ViT骨干网络的基线,提高了每个骨干网络上的跨任务平均准确率和最差任务准确率。

英文摘要:

Model merging offers a promising avenue for knowledge integration and parallel development without retraining. Yet, existing methods either ignore the geometry of the loss landscape or rely on intractable full-space Hessian approximations. We propose EpiMer, a framework that casts model merging as solving the Fréchet mean on a Riemannian manifold and restricts the computation to a low-rank subspace spanned by the task vectors. With the expected Hessian as the metric, we reveal a connection between local curvature and epistemic uncertainty of the parameters. Our theoretical analysis decomposes the merging error bound into the subspace Fréchet variance and the residual energy, and provides a closed-form characterization of when curvature-aware merging provably outperforms flat-geometry methods. In addition, our framework unifies both curvature-aware methods and recent spectral methods as special cases of the subspace Fréchet mean with different geometric metrics. Merging fine-tuned CLIP-ViT models on eight image classification tasks, Epistemic Merging strictly outperforms the baselines on all three CLIP-ViT backbones at matched rank, improving the across-task average accuracy and worst-task accuracy on every backbone.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑