用于模型合并的基于因果干预的激活参数定位
Activated Parameter Locating via Causal Intervention for Model Merging
AI总结：
针对模型合并中参数冗余冲突问题，提出基于因果干预的激活参数定位方法APL，结合梯度近似策略，更精准地丢弃参数缓解冲突，在域内外场景均验证了有效性。
摘要：
模型合并将多个同源模型整合为一个模型，无需额外训练即可实现令人信服的泛化能力。该问题的核心挑战是解决多个模型间的参数冗余与冲突。现有研究表明，丢弃部分delta参数可在维持性能的同时缓解冲突，但这些方法通常随机丢弃或基于参数量级丢弃，忽略了微调模型中蕴含的任务特定信息。本文提出激活参数定位（Activated Parameter Locating, APL）方法，利用因果干预评估参数重要性，实现更精准的参数丢弃与更优的冲突缓解。此外，为降低大量参数划分带来的计算复杂度，我们还为APL引入了有理论支撑的梯度近似策略。在域内和域外设置下开展的模型合并实验及相关分析，证明了APL的有效性。
英文摘要：
Model merging combines multiple homologous models into one model, achieving convincing generalization without the necessity of additional training. A key challenge in this problem is resolving parameter redundancies and conflicts across multiple models. Existing models have demonstrated that dropping a portion of delta parameters can alleviate conflicts while maintaining performance. However, these methods often drop parameters either randomly or based on magnitude, overlooking task-specific information embedded in fine-tuned models. In this paper, we propose an Activated Parameter Locating (APL) method that utilizes causal intervention to estimate parameter importance, enabling more precise parameter drops and better conflict mitigation. Moreover, to reduce the computational complexity associated with a large number of parameter partitions, we also introduce a theoretically supported gradient approximation strategy for APL. Experiments on model merging within both in-domain and out-of-domain settings, along with associated analyses, showcase the effectiveness of APL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
