Cendol：面向印尼语言的开源指令微调生成式大语言模型
Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages
AI总结：
针对低资源印尼语言大模型质量不足的问题，研究推出多架构多规模的印尼语指令微调模型Cendol，在多样任务上提升20%性能，提出词汇适配提升效率，并验证预训练安全性可跨语言迁移。
摘要：
大语言模型（LLM）在多个领域和语言中展现出类人的卓越能力，但在低资源语言（如印尼本土语言）中存在明显的质量差距，导致其在这类语言环境中效果不佳、效率低下。为弥补这一质量差距，我们推出Cendol——一系列印尼语大语言模型，涵盖仅解码器和编码器-解码器两种架构，覆盖多种模型规模。我们验证了Cendol在多样任务中的有效性，性能提升达20%，并证明其能够泛化到未见过的任务和印尼本土语言中。此外，尽管Cendol模型在掌握印尼本土知识与文化价值方面存在局限，但仍展现出更高的人类偏好度。我们还讨论了LoRA等参数高效微调方法在语言适配中的不足，转而提出使用词汇适配来提升效率。最后，我们评估了Cendol的安全性，结果表明：即便没有RLHF和安全微调，单一语言（如英语）预训练中的安全性也能迁移到印尼语等低资源语言中。
英文摘要：
Large language models (LLMs) show remarkable human-like capability in various domains and languages. However, a notable quality gap arises in low-resource languages, e.g., Indonesian indigenous languages, rendering them ineffective and inefficient in such linguistic contexts. To bridge this quality gap, we introduce Cendol, a collection of Indonesian LLMs encompassing both decoder-only and encoder-decoder architectures across a range of model sizes. We highlight Cendol's effectiveness across a diverse array of tasks, attaining 20% improvement, and demonstrate its capability to generalize to unseen tasks and indigenous languages of Indonesia. Furthermore, Cendol models showcase improved human favorability despite their limitations in capturing indigenous knowledge and cultural values in Indonesia. In addition, we discuss the shortcomings of parameter-efficient tunings, such as LoRA, for language adaptation. Alternatively, we propose the usage of vocabulary adaptation to enhance efficiency. Lastly, we evaluate the safety of Cendol and showcase that safety in pre-training in one language such as English is transferable to low-resource languages, such as Indonesian, even without RLHF and safety fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
