Sailor：面向东南亚地区的开放语言模型
Sailor: Open Language Models for South-East Asia
AI总结：
Sailor 是基于 Qwen1.5 持续预训练的 0.5B 至 7B 开放语言模型系列，面向东南亚多语言场景，通过 BPE dropout、数据清洗去重和代理模型优化数据混合，在常识推理、问答、阅读理解和考试等任务上表现优异。
摘要：
我们提出了 Sailor，这是一个参数量从 0.5B 到 7B 的开放语言模型系列，专为东南亚（SEA）语言量身定制。这些模型基于 Qwen1.5 进行持续预训练，Qwen1.5 是一个在多语言使用场景中表现出色的语言模型。Sailor 模型从 Qwen1.5 开始接收 200B 到 400B 个 token，主要涵盖英语、中文、越南语、泰语、印度尼西亚语、马来语和老挝语。训练过程中采用了多种技术，包括用于提高模型鲁棒性的 BPE dropout、激进的数据清洗与去重，以及用于优化数据混合的小型代理模型。在四个典型任务上的实验结果表明，Sailor 模型在不同基准测试中展现出强大的性能，包括常识推理、问答、阅读理解和考试。秉持开源精神，我们通过本报告分享我们的见解，以激发更广泛的兴趣，推动面向多语言使用场景的大语言模型开发。
英文摘要：
We present Sailor, a family of open language models ranging from 0.5B to 7B parameters, tailored for South-East Asian (SEA) languages. These models are continually pre-trained from Qwen1.5, a great language model for multilingual use cases. From Qwen1.5, Sailor models accept 200B to 400B tokens, primarily covering the languages of English, Chinese, Vietnamese, Thai, Indonesian, Malay, and Lao. The training leverages several techniques, including BPE dropout for improving the model robustness, aggressive data cleaning and deduplication, and small proxy models to optimize data mixture. Experimental results on four typical tasks indicate that Sailor models demonstrate strong performance across different benchmarks, including commonsense reasoning, question answering, reading comprehension and examination. Embracing the open-source spirit, we share our insights through this report to spark a wider interest in developing large language models for multilingual use cases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
