arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.00738cs.CL
收藏

SeaLLMs——面向东南亚的大语言模型

SeaLLMs -- Large Language Models for Southeast Asia

DAMO Academy, Alibaba Group(达摩院(阿里巴巴集团))
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng … 展开作者

Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, Lidong Bing

AI总结:

本文提出面向东南亚语言的SeaLLMs系列模型,基于Llama-2进行持续预训练与指令对齐,在泰语、高棉语等非拉丁语言上超越ChatGPT-3.5,同时保持轻量高效。

摘要:

尽管大语言模型(LLMs)在各类任务中取得了显著成就,但仍存在偏向英语等高资源语言的语言偏见,往往以牺牲低资源语言和区域语言为代价。为解决这一不平衡问题,我们提出了SeaLLMs,这是一个专门针对东南亚(SEA)语言的创新语言模型系列。SeaLLMs基于Llama-2模型构建,并通过扩展词表的持续预训练、专门的指令与对齐微调进一步提升,以更好地捕捉区域语言的细微之处。这使它们能够尊重并反映当地文化规范、习俗、风格偏好和法律考量。我们的综合评估表明,SeaLLM-13b模型在广泛的 linguistic 任务和助手式指令遵循能力上,相较于同类开源模型表现出更优越的性能。此外,它们在泰语、高棉语、老挝语和缅甸语等非拉丁语言上大幅超越ChatGPT-3.5,同时保持轻量化和成本效益高的运行特性。

英文摘要:

Despite the remarkable achievements of large language models (LLMs) in various tasks, there remains a linguistic bias that favors high-resource languages, such as English, often at the expense of low-resource and regional languages. To address this imbalance, we introduce SeaLLMs, an innovative series of language models that specifically focuses on Southeast Asian (SEA) languages. SeaLLMs are built upon the Llama-2 model and further advanced through continued pre-training with an extended vocabulary, specialized instruction and alignment tuning to better capture the intricacies of regional languages. This allows them to respect and reflect local cultural norms, customs, stylistic preferences, and legal considerations. Our comprehensive evaluation demonstrates that SeaLLM-13b models exhibit superior performance across a wide spectrum of linguistic tasks and assistant-style instruction-following capabilities relative to comparable open-source models. Moreover, they outperform ChatGPT-3.5 in non-Latin languages, such as Thai, Khmer, Lao, and Burmese, by large margins while remaining lightweight and cost-effective to operate.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑