MERaLiON-TextLLM:大语言模型在中文、印度尼西亚语、马来语和新加坡英语中的跨语言理解
MERaLiON-TextLLM: Cross-Lingual Understanding of Large Language Models in Chinese, Indonesian, Malay, and Singlish
AI总结:
本文提出MERaLiON-TextLLM系列开源模型,基于Llama-3-8B-Base通过持续预训练和权重合并,提升中文、印度尼西亚语、马来语和新加坡英语的理解与生成,超越官方Llama-3模型。
摘要:
多语言大语言模型(MLLMs)在各种语言中展现出了令人印象深刻的能力。然而,不同语系之间的效能可能存在很大差异,尤其是对于语言资源有限的语系。本报告介绍了MERaLiON-TextLLM,一系列专门为提升中文、印度尼西亚语、马来语和新加坡英语的理解与生成能力而设计的开源语言模型。初始发布的模型基于Llama-3-8B-Base构建,并通过精心设计的持续预训练和权重合并流程进行优化。我们的方法在这些语言的基准测试中实现了性能提升,超越了官方Llama-3模型的能力。我们提供模型检查点作为资源,以支持跨语言理解领域的进一步研究与发展。
英文摘要:
Multilingual large language models (MLLMs) have shown impressive capabilities across a variety of languages. However, efficacy can differ greatly between different language families, especially for those with limited linguistic resources. This report presents MERaLiON-TextLLM, a series of open-source language models specifically tailored to improve understanding and generation in Chinese, Indonesian, Malay, and Singlish. The initial released model is built on Llama-3-8B-Base and refined through a meticulously crafted process of continued pre-training and weight merging. Our approach achieves performance improvements across benchmarks in these languages, exceeding the capabilities of the official Llama-3 models. We provide the model checkpoints as a resource to support further research and development in cross-lingual language understanding.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。