AI 中文总结
CORAM通过将目标矩阵划分为行切片并在流形上合并任务因子，结合放大系数、扩散切片与残差通路，在多套件中较OrthoMerge提升0.25-1.35点，性能匹配或超越最强权重空间基线。
AI 中文摘要
合并微调后的模型可在无需联合训练或访问原始数据的情况下整合专门能力。多数方法在欧几里得权重空间中通过线性算术操作，无法保留更新的几何结构。正交模型合并（OrthoMerge）为每个权重矩阵使用单一正交变换，但此类变换无法改变奇异值。我们提出CORAM，它将每个目标矩阵划分为行切片，在对应的基础模型SVD框架中通过奇异值分解表示每个专家切片，并在对应流形上合并特定任务因子。由于流形平均会收缩合并后的更新，CORAM应用放大系数λ=κĉ。尺度ĉ由专家和合并更新的范数估计，对于N个更新幅度相当的专家，ĉ约为√N。恢复强度κ根据专家更新的离散度选择，无需评估候选合并模型，该规则在所有评估套件上与最优扫描值的差距保持在0.72点以内。CORAM还包含扩散切片以将高度更新的行分布到各切片，以及针对非目标层的残差通路。在覆盖三个模型系列、3B至9B规模及语言和视觉-语言专家的四个套件中，CORAM较OrthoMerge提升了0.25至1.35点，且匹配或超越了最强的权重空间基线。
英文摘要
Merging finetuned models combines specialized capabilities without joint training or access to the original data. Most methods operate by linear arithmetic in Euclidean weight space, which cannot carry the geometry of the update. Orthogonal Model Merging (OrthoMerge) uses a single orthogonal transform for each weight matrix, but such a transform cannot change singular values. We propose CORAM, which partitions each target matrix into row slices, represents every expert slice by its singular value decomposition in the corresponding base-model SVD frame, and merges the task-specific factors on their corresponding manifolds. Because manifold averaging contracts the merged update, CORAM applies an amplification coefficient $λ=κ\hat{c}$. The scale c_hat is estimated from the expert and merged update norms and is approximately $\sqrt{N}$ for $N$ experts with comparable update magnitudes. The restoration strength kappa is selected from the dispersion of expert updates without evaluating candidate merged models. This rule remains within 0.72 points of the best swept value on all evaluated suites. CORAM also includes spread slicing to distribute highly updated rows across slices and a residual pathway for non-target layers. Across four suites covering three model families, 3B to 9B scales, and language and vision-language experts, CORAM improves over OrthoMerge by 0.25 to 1.35 points and matches or exceeds the strongest weight-space baselines.
Comments26 pages, including supplementary material
