通过源保护更新缓解LLM目标语言适应中的灾难性遗忘
Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates
AI总结：
本文提出源保护更新策略，利用少量源数据和参数重要性评分，在低资源条件下有效缓解LLM在目标语言适应中的灾难性遗忘问题，实验表明其在保持源语言能力的同时提升了目标语言性能。
摘要：
扩展指令型大语言模型（LLM）的语言多样性对于全球普及至关重要，但通常受到依赖昂贵的专门目标语言标记数据和适应过程中的灾难性遗忘的阻碍。本文在低资源约束下，提出了一种基于源保护更新（SSU）的策略，通过选择性参数更新策略主动保留源知识。利用少量源数据和参数重要性评分方法，SSU识别出维持源能力的关键参数，并通过列级冻结策略在适应前保护这些参数。在五种语言类型多样和7B、13B模型上的实验表明，SSU有效缓解了灾难性遗忘，将单语源任务性能下降降至平均3.4%（7B）和2.8%（13B），远低于全微调的20.3%和22.3%。SSU在目标语言性能上与全微调竞争，7B模型在所有基准上均优于全微调，13B模型在多数基准上亦优于全微调。
英文摘要：
Expanding the linguistic diversity of instruct large language models (LLMs) is crucial for global accessibility but is often hindered by the reliance on costly specialized target language labeled data and catastrophic forgetting during adaptation. We tackle this challenge under a realistic, low-resource constraint: adapting instruct LLMs using only unlabeled target language data. We introduce Source-Shielded Updates (SSU), a selective parameter update strategy that proactively preserves source knowledge. Using a small set of source data and a parameter importance scoring method, SSU identifies parameters critical to maintaining source abilities. It then applies a column-wise freezing strategy to protect these parameters before adaptation. Experiments across five typologically diverse languages and 7B and 13B models demonstrate that SSU successfully mitigates catastrophic forgetting. It reduces performance degradation on monolingual source tasks to just 3.4% (7B) and 2.8% (13B) on average, a stark contrast to the 20.3% and 22.3% from full fine-tuning. SSU also achieves target-language performance highly competitive with full fine-tuning, outperforming it on all benchmarks for 7B models and the majority for 13B models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
