正交但耦合：解耦几何组件用于模型合并
Orthogonal Yet Coupled: Decoupling Geometric Components for Model Merging
- Northeastern University（东北大学）
- CIS, LMU Munich（慕尼黑大学 CIS）
- Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
- Shanghai Jiao Tong University（上海交通大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
提出DiGA框架，通过正交分解任务向量为几何组件并独立聚合，避免跨组件耦合，提升模型合并性能并减少能力退化。
AI中文摘要：
合并预训练模型已成为将多种能力整合到单一统一模型中的有效方法。然而，现有的合并方法通常将每个任务向量视为不可分割的合并单元，忽略了其中编码的异构几何变化。这种处理方式可能引发跨组件耦合：当合并决策基于完整任务向量的统计信息时，一个组件的几何特征可能影响另一个组件的选择、加权或组合方式，从而可能降低合并模型的质量。为解决这一问题，我们提出了DiGA，一种解耦的几何感知模型合并框架。利用预训练权重作为共享的几何参考，DiGA将每个任务向量正交分解为对应于不同几何属性的组件。DiGA不是将任务向量作为整体进行合并，而是在各自子空间内独立聚合对应的组件，随后将其重新组合为统一的更新。这种逐组件的公式保留了每个组件的几何身份，并防止一个组件的特征干扰另一个组件的聚合。此外，DiGA可以集成到广泛的现有模型合并方法中。跨多种模型、任务和合并方法的广泛实验表明，DiGA提高了合并模型的性能并减少了能力退化。我们的代码库位于此https URL。
英文摘要：
Merging pretrained models has emerged as an effective approach for consolidating diverse capabilities into a single unified model. However, prevailing merging methods typically treat each task vector as an indivisible merging unit, overlooking the heterogeneous geometric changes encoded within it. This treatment can induce cross-component coupling: when merging decisions are derived from statistics of the complete task vector, the geometric characteristics of one component may influence how another is selected, weighted, or combined, potentially degrading the quality of the merged model. To address this issue, we propose DiGA, a Disentangled Geometry-Aware model merging framework. Using the pretrained weights as a shared geometric reference, DiGA orthogonally decomposes each task vector into components corresponding to distinct geometric attributes. Rather than merging the task vectors as a whole, DiGA aggregates corresponding components independently within their respective subspaces and subsequently recombines them into a unified update. This component-wise formulation preserves the geometric identity of each component and prevents the characteristics of one component from interfering with the aggregation of another. Furthermore, DiGA can be incorporated into a broad range of existing model merging methods. Extensive experiments across diverse models, tasks, and merging methods demonstrate that DiGA improves merged-model performance and reduces capability degradation. Our repository is on https://github.com/wzj1718/DiGA.
