arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.03994cs.CLcs.AI
收藏

释放模型合并技术在低资源语言中的潜力

Unlocking the Potential of Model Merging for Low-Resource Languages

Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Center for Data Science, Peking University(北京大学数据科学中心);College of Engineering, Peking University(北京大学工学院)

Mingxu Tao, Chen Zhang, Quzhe Huang, Tianyao Ma, Songfang Huang, Dongyan Zhao, Yansong Feng

AI总结:

本文提出用模型合并替代“先持续预训练后监督微调”的方法,在无目标语言SFT数据的情况下为低资源语言构建具备任务求解能力的LLMs,并通过引入松弛变量缓解性能饱和问题,实验证明其在数据极度稀缺场景下优于传统方法。

摘要:

将大语言模型(LLMs)适配到新语言通常需要先进行持续预训练(CT),再进行监督微调(SFT)。然而,这种“先CT后SFT”的方法在低资源语言场景下受限于数据不足,难以兼顾语言建模能力和任务求解能力。因此,我们提出将模型合并作为低资源语言的一种替代方案,在无需额外训练的情况下,将具有不同能力的模型组合成单一模型。我们利用模型合并技术,在没有目标语言SFT数据的情况下,为低资源语言开发具备任务求解能力的LLMs。我们基于Llama-2-7B的实验表明,模型合并能够有效赋予低资源语言LLMs任务求解能力,在数据极度稀缺的场景下优于“先CT后SFT”的方法。我们观察到,随着训练token数量增加,模型合并会出现性能饱和现象,因此进一步分析了合并过程,并在模型合并算法中引入松弛变量,以减轻重要参数的损失,从而提升性能。我们希望模型合并凭借其更高的数据效率,能够惠及更多面临数据稀缺问题的人类语言。

英文摘要:

Adapting large language models (LLMs) to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT). However, this CT-then-SFT approach struggles with limited data in the context of low-resource languages, failing to balance language modeling and task-solving capabilities. We thus propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. We use model merging to develop task-solving LLMs for low-resource languages without SFT data in the target languages. Our experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data. Observing performance saturation in model merging with more training tokens, we further analyze the merging process and introduce a slack variable to the model merging algorithm to mitigate the loss of important parameters, thereby enhancing performance. We hope that model merging can benefit more human languages suffering from data scarcity with its higher data efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑