通过跨语言知识迁移教授 Llama 一门新语言
Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer
AI总结：
本文面向低资源爱沙尼亚语，基于 Llama 2 结合单语预训练与跨语言指令微调，推出首个开源指令模型 Llammas 和数据集 Alpaca-est。
摘要：
本文探索以低成本方法使预训练大语言模型（LLM）适应新的低资源语言，特别聚焦爱沙尼亚语。我们利用 Llama 2 模型，研究将跨语言指令微调与额外单语预训练相结合的影响。结果表明，即使只进行相对少量的额外单语预训练，随后再进行跨语言指令微调，也能显著提升在爱沙尼亚语上的表现。此外，我们展示了从高质量英语指令到爱沙尼亚语的跨语言知识迁移，从而提升了常识推理和多轮对话能力。我们的最佳模型名为 Llammas，是首个面向爱沙尼亚语的开源指令遵循 LLM。同时，我们发布了 Alpaca-est，这是爱沙尼亚首个通用任务指令数据集。这些贡献标志着在为爱沙尼亚语开发开源 LLM 方向上的初步进展。
英文摘要：
This paper explores cost-efficient methods to adapt pretrained Large Language Models (LLMs) to new lower-resource languages, with a specific focus on Estonian. Leveraging the Llama 2 model, we investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining. Our results demonstrate that even a relatively small amount of additional monolingual pretraining followed by cross-lingual instruction-tuning significantly enhances results on Estonian. Furthermore, we showcase cross-lingual knowledge transfer from high-quality English instructions to Estonian, resulting in improvements in commonsense reasoning and multi-turn conversation capabilities. Our best model, named \textsc{Llammas}, represents the first open-source instruction-following LLM for Estonian. Additionally, we publish Alpaca-est, the first general task instruction dataset for Estonia. These contributions mark the initial progress in the direction of developing open-source LLMs for Estonian.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
