发表机构
Pengcheng Laboratory; City University of Hong Kong; Southern University of Science and Technology; Harbin Institute of Technology; Northwestern Polytechnical University（鹏城实验室; 香港城市大学; 南方科技大学; 哈尔滨工业大学; 西北工业大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出贡献感知结构化稀疏性（CASS），通过识别并保留任务相关组件（注意力头和FFN神经元）减少模型合并中的参数干扰，并在视觉和语言基准上提升多种合并基线性能。
AI 中文摘要
模型合并将针对特定任务微调的模型整合为单一的多任务模型，但常常因任务向量更新相互冲突而遭受参数干扰。现有方法通常基于权重幅度或随机启发式剪枝任务向量来缓解冲突，将Transformer视为非结构化的“参数包”，忽视了其固有的模块化特性。本文提出贡献感知结构化稀疏性（CASS），一种统一框架，通过识别并保留任务特定组件来减少参数干扰。CASS的核心是贡献感知结构化掩码，用于识别与任务相关的注意力头和前馈网络神经元。我们在两种设置中实例化该掩码：CASS-Merging，主要的后处理设置，其中掩码作为现有合并算子的即插即用去噪滤波器；以及CASS-Tuning，一种扩展，适用于具有微调访问权限的场景，其中掩码约束梯度以减少任务向量之间的结构重叠。我们的分析表明，任务相关组件是稀疏且部分不相交的，支持结构化组件级过滤作为减少合并干扰的有效方法。在视觉（ViT，20个任务）和语言（RoBERTa，8个任务；Qwen2.5，4个任务）基准上的广泛实验表明，CASS改进了多种代表性合并基线。
英文摘要
Model merging integrates task-specific fine-tuned models into a single multi-task model, but often suffers from parameter interference caused by conflicting task-vector updates. Existing methods typically mitigate conflicts by pruning task vectors based on weight magnitude or random heuristics, treating Transformers as unstructured ``bags of parameters'' and overlooking their inherent modularity. In this paper, we propose \textbf{C}ontribution-\textbf{A}ware \textbf{S}tructured \textbf{S}parsity (CASS), a unified framework that reduces parameter interference by identifying and preserving task-specific components. At the core of CASS is a contribution-aware structured mask that identifies task-relevant attention heads and FFN neurons. We instantiate this mask in two settings: CASS-Merging, the primary post-hoc setting where masks serve as a plug-and-play denoising filter for existing merging operators, and CASS-Tuning, an extension for scenarios with fine-tuning access where masks constrain gradients to reduce structural overlap between task vectors. Our analysis shows that task-relevant components are sparse and partially disjoint, supporting structured component-level filtering as an effective way to reduce merging interference. Extensive experiments across vision (ViT, 20 tasks) and language (RoBERTa, 8 tasks; Qwen2.5, 4 tasks) benchmarks demonstrate that CASS improves a range of representative merging baselines.
CommentsAccepted to NeurIPS 2026
