通过从资源丰富语言的自蒸馏增强大型语言模型的多语言能力
Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages
AI总结：
本文提出SDRRL方法，通过从资源丰富语言自蒸馏并利用LLMs内部能力，有效提升多语言性能，同时最小化对原始语言性能的影响。
摘要：
尽管大型语言模型（LLMs）已在多语言语料库上进行了预训练，但与少数资源丰富语言相比，它们在大多数语言上的性能仍然落后。缓解此问题的一种常见方法是将训练数据从资源丰富语言翻译成其他语言，然后继续训练。然而，仅依赖翻译获取数据而忽略LLMs跨语言的原始能力并不总是有效的，我们表明这将限制跨语言知识转移的性能。在本研究中，我们提出SDRRL，一种基于从资源丰富语言自蒸馏的方法，通过利用LLMs在资源丰富语言上的内部能力来有效提升多语言性能。我们在不同LLMs（LLaMA-2和SeaLLM）和源语言上，于多种理解与生成任务中进行评估，实验结果表明，SDRRL能显著增强多语言能力，同时将对资源丰富语言原始性能的影响降至最低。
英文摘要：
While large language models (LLMs) have been pre-trained on multilingual corpora, their performance still lags behind in most languages compared to a few resource-rich languages. One common approach to mitigate this issue is to translate training data from resource-rich languages into other languages and then continue training. However, using the data obtained solely relying on translation while ignoring the original capabilities of LLMs across languages is not always effective, which we show will limit the performance of cross-lingual knowledge transfer. In this work, we propose SDRRL, a method based on Self-Distillation from Resource-Rich Languages that effectively improve multilingual performance by leveraging the internal capabilities of LLMs on resource-rich languages. We evaluate on different LLMs (LLaMA-2 and SeaLLM) and source languages across various comprehension and generation tasks, experimental results demonstrate that SDRRL can significantly enhance multilingual capabilities while minimizing the impact on original performance in resource-rich languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
