Compass：面向东南亚的大型多语言语言模型
Compass: Large Multilingual Language Model for South-east Asia
AI总结：
针对东南亚低资源语言大模型效果不佳的问题，提出专为东南亚定制的多语言大模型CompassLLM，通过多阶段课程预训练、指令微调与DPO对齐，在多任务上超越多个基准模型。
摘要：
大语言模型在英语、汉语等语言资源丰富的语言上已展现出出色能力，但在资源匮乏的语言（尤其是东南亚地区的语言，如印度尼西亚语）中，其效果会显著下降。这类语言的语料资源稀缺，带来了训练不充分、词汇覆盖范围有限、评估流程困难等挑战。针对这些需求，我们提出了CompassLLM，这是一款专门为东南亚语言定制的大型多语言模型，核心目标是支撑Shopee的发展需求。我们的方法包含多项关键策略：为逐步提升多语言能力，我们采用结合课程学习的多阶段预训练策略，逐步加大对低资源语言的训练比重；同时，为更好地适配低资源语言的人类指令，我们整理并生成了高质量的多语言人类指令库，通过有监督指令微调得到CompassLLM-SFT模型；最后，为强化模型与人类偏好行为的对齐，我们采用直接偏好优化（Direct Preference Optimization, DPO）方法，得到CompassLLM-DPO模型。对CompassLLM的初步评估取得了令人振奋的结果：自动化评估与人工评估均证实，我们的模型在各类评估任务上超越了Vicuna-7b-v1.5、Sealion、Falcon、SeaLLM等基准模型，尤其在印度尼西亚语等东南亚语言上表现出更优的性能。
英文摘要：
Large language models have exhibited significant proficiency in languages endowed with extensive linguistic resources, such as English and Chinese. Nevertheless, their effectiveness notably diminishes when applied to languages characterized by limited linguistic resources, particularly within the Southeast Asian linguistic landscape, such as Indonesian. The scarcity of linguistic resources for these languages presents challenges associated with inadequate training, restricted vocabulary coverage, and challenging evaluation processes. In response to these exigencies, we have introduced CompassLLM, a large multilingual model specifically tailored for Southeast Asian languages, with the primary aim of supporting the developmental requirements of Shopee. Our methodology encompasses several key strategies. To progressively enhance multilingual proficiencies, we implemented a multi-stage pre-training strategy integrated with curriculum learning, gradually intensifying the focus on low-resource languages. Concurrently, to better accommodate low-resource human instructions, we curated and generated a repository of high-quality multilingual human instructions, culminating the CompassLLM-SFT model through supervised instruction fine-tuning. Finally, to reinforce the model's alignment with human preference behaviors, we have embraced the principle of Direct Preference Optimization (DPO) to obtain CompassLLM-DPO model. Preliminary evaluation of the CompassLLM model yields promising results, with our model surpassing benchmark models like Vicuna-7b-v1.5, Sealion, Falcon and SeaLLM, across diverse evaluation tasks, as verified through both automated and human-driven assessments. Notably, our model exhibits its superior performance in South-east Asia languages, such as Indonesian language.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
