AI 中文总结
针对密集专家合并的跨专家冲突容量分配等问题，提出SigMerge框架，在21种配对设置中优于三类基线合并方法，平均提升15.0%且获最佳平均排名。
AI 中文摘要
密集专家合并将领域专用语言模型组合成单个检查点，通常通过在权重空间中接纳任务向量支持来实现。然而，这种接纳受三个决策支配，现有方法仅部分解决：从跨专家冲突中开放层容量的位置、基于领域需求谁应占用该容量、以及如何在不依赖代价高昂的方案搜索的情况下接纳所得支持。为解决这些问题，我们提出SigMerge（签名引导的容量占用），这是一种用于密集专家合并的结构化容量分配框架。从密集基础合并开始，冲突签名确定跨专家冲突产生的每一层容量，正基础合并赤字确定每个领域对该容量的份额，而顺序占用规则按所得层-领域预算接纳每个专家增量。在涵盖7个密集基础合并和3个模型池的21种配对设置中，SigMerge在所有设置上均实现改进（平均提升15.0%），并在6种合并方法中取得最佳平均排名（1.67），优于三类合并基线方法。
英文摘要
Dense expert merging combines domain-specialized language models into one single checkpoint, typically by admitting task-vector support in weight space. However, this admission is governed by three decisions that existing methods answer only partially: where to open layer capacity from cross-expert conflict, who should occupy that capacity based on domain demand, and how to admit the resulting support without relying on costly recipe search. To tackle these issues, we propose SigMerge (Signature-Guided Capacity Occupancy), a structured capacity assignment framework for dense expert merging. Starting from a dense base merge, conflict signatures set each layer's capacity from cross-expert conflict, positive base-merge deficits set each domain's share of that capacity, and a sequential occupancy rule admits each expert delta up to the resulting layer-domain budget. Across 21 paired settings spanning seven dense base merges and three model pools, SigMerge improves every one (by 15.0% on average) and achieves the best average rank (1.67) among six merging methods, outperforming three categories of merging baselines.
Comments31 pages, 20 figures
