参数对齐减轻多语言专家语言模型中的灾难性遗忘
Specializing Without Forgetting: Analyzing Knowledge Preservation in Multilingual Model Adaptation
AI总结：
针对多语言持续预训练中的灾难性遗忘问题，提出五种层感知参数对齐策略（硬冻结、软正则化、事后权重恢复和模型合并），在32种训练语言和保留语言上评估，证明参数对齐能有效减少遗忘且语言获取成本低。
摘要：
虽然持续预训练（CPT）是将大型语言模型扩展到新语言的一种实用方法，但在目标数据上的简单微调会通过灾难性侵蚀现有能力。围绕语系组织训练可以减少跨语言干扰，但无法单独防止下游任务所需通用知识的遗忘。我们将这种遗忘与多语言CPT中的参数漂移联系起来，并提出了一套五种层感知参数对齐策略：硬层冻结、软正则化、事后权重恢复和模型合并。我们在涵盖五个语系32种训练语言以及保留语言的基准上，沿四个评估轴（困惑度、阅读理解、物理推理和翻译）系统地将我们的对齐策略与两个无正则化的CPT基线进行比较。参数对齐在语言获取成本最小的情况下显著减少了遗忘：层冻结和正则化最好地保留了理解能力，而事后恢复则带来了最强的翻译提升。总之，这些结果描绘了族专家CPT的获取-遗忘边界，并为每种策略提供了与其最佳服务任务配对的实用部署指南。
英文摘要：
While continual pretraining (CPT) is a practical way to extend large language models to new languages, naïve finetuning often erodes existing capabilities through catastrophic forgetting. We investigate which model layers drive this trade-off, and whether interventions at these layers can guide knowledge preservation during adaptation. We interpolate gemma-3-4b model states before and after CPT on five language families to localize forgetting on reading comprehension and translation, finding that middle-layer reversion yields the largest comprehension recovery, while translation effects vary by language family and direction. Guided by these findings, we evaluate CPT strategies that leverage this layer information to mitigate forgetting: layer freezing, layer-range L2 regularization, post-hoc layer reversion, and model souping, comparing all strategies against joint multilingual and family-specific vanilla CPT baselines. We find that preserving the layer weights identified via model interpolation substantially reduces comprehension loss relative to joint CPT, with layer freezing exceeding base model performance on average. However, these strategies yield mixed translation results: dense training or post-hoc reversion often outperforms both training-time constraints and family-specific specialization, complicating prior assumptions about how models should be aligned when extended to new tasks. Instead, we argue that multilingual adaptation strategy should be informed by target language, base model knowledge, and downstream task, and propose interpolation-based localization as a diagnostic for identifying candidate layers before committing to a training-time intervention in a new setting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
