发表机构
KAIST; Kakao（韩国科学技术院; Kakao公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对多目标统一模型后训练易受顺序影响及朴素合并失效的问题，提出混合训练合并（MTM），通过目标偏置数据混合训练分支并迭代优化，在代码、数学等任务上优于朴素合并并保持行为分离。
AI 中文摘要
统一语言模型越来越被期望在单组参数内整合异构能力，如数学、代码、指令遵循和可控的思考行为。常见的解决方案是在多个目标上进行顺序后训练，但这会将所有目标纠缠在一条优化轨迹上，并使最终模型对训练顺序、数据比例、调度和停止标准高度敏感。权重空间合并提供了一种模块化替代方案，但朴素地合并单目标专家常常失败：领域能力急剧退化，或思考/非思考模式坍缩为一种主导行为。我们将这两种失败归因于不兼容的权重空间几何：在单一目标上训练的专家会漂移到参数空间的遥远区域，使其插值落在任何共享低损失盆地之外。我们提出混合训练合并（MTM），该方法让每个分支在目标偏置的数据混合上训练，而非单一目标，使其暴露于跨目标交互，并在合并时使分支兼容。MTM使用合并模型的评估作为选择分支混合的低成本信号，避免昂贵的数据混合消融。该过程是迭代的：每轮使用全局选择的合并系数提升基础模型，并在保留其他目标的约束下，使用领域偏好系数细化每个分支混合。为扩展到单纯形网格搜索之外，MTM使用基于qNEHVI的多目标贝叶斯优化。在代码、数学、指令遵循和思考/非思考控制方面，MTM优于朴素合并，并在单目标合并坍缩之处保持行为分离，表明有效的统一模型需要训练分支使其可合并。
英文摘要
Unified language models are increasingly expected to combine heterogeneous capabilities, such as mathematics, code, instruction following, and controllable thinking behavior, within a single set of parameters. A common solution is sequential post-training on multiple objectives, but this entangles all objectives along one optimization trajectory and makes the final model highly sensitive to training order, data ratios, schedules, and stopping criteria. Weight-space merging offers a modular alternative, but naive merging of single-objective experts often fails: domain capabilities degrade sharply, or think/non-think modes collapse into one dominant behavior. We attribute both failures to incompatible weight-space geometry: experts trained on single objectives drift to distant regions of parameter space, placing their interpolations outside any shared low-loss basin. We propose Mixture-Trained Merging (MTM), which trains each branch on an objective-biased data mixture rather than a single objective, exposing it to cross-objective interactions and making branches compatible at merge time. MTM uses merged-model evaluations as a low-cost signal for selecting branch mixtures, avoiding expensive data-mixture ablations. The procedure is iterative: each round promotes the base model using globally selected merge coefficients and refines each branch mixture using domain-preferred coefficients under constraints that preserve other objectives. To scale beyond simplex grid search, MTM uses qNEHVI-based multi-objective Bayesian optimization. Across code, mathematics, instruction following, and think/non-think control, MTM outperforms naive merging and preserves behavioral separation where single-objective merging collapses, suggesting that effective unified models require branches trained to be mergeable.
CommentsAccepted at NeurIPS 2026
