arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.08335cs.CLcs.AI
收藏

MERaLiON-TextLLM:大语言模型在中文、印度尼西亚语、马来语和新加坡英语中的跨语言理解

MERaLiON-TextLLM: Cross-Lingual Understanding of Large Language Models in Chinese, Indonesian, Malay, and Singlish

Institute for Infocomm Research (I2 R), A*STAR(新加坡科研局信息通信研究所)

Xin Huang, Tarun Kumar Vangani, Minh Duc Pham, Xunlong Zou, Bin Wang, Zhengyuan Liu, Ai Ti Aw

AI总结:

本文提出MERaLiON-TextLLM系列开源模型,基于Llama-3-8B-Base通过持续预训练和权重合并,提升中文、印度尼西亚语、马来语和新加坡英语的理解与生成,超越官方Llama-3模型。

摘要:

多语言大语言模型(MLLMs)在各种语言中展现出了令人印象深刻的能力。然而,不同语系之间的效能可能存在很大差异,尤其是对于语言资源有限的语系。本报告介绍了MERaLiON-TextLLM,一系列专门为提升中文、印度尼西亚语、马来语和新加坡英语的理解与生成能力而设计的开源语言模型。初始发布的模型基于Llama-3-8B-Base构建,并通过精心设计的持续预训练和权重合并流程进行优化。我们的方法在这些语言的基准测试中实现了性能提升,超越了官方Llama-3模型的能力。我们提供模型检查点作为资源,以支持跨语言理解领域的进一步研究与发展。

英文摘要:

Multilingual large language models (MLLMs) have shown impressive capabilities across a variety of languages. However, efficacy can differ greatly between different language families, especially for those with limited linguistic resources. This report presents MERaLiON-TextLLM, a series of open-source language models specifically tailored to improve understanding and generation in Chinese, Indonesian, Malay, and Singlish. The initial released model is built on Llama-3-8B-Base and refined through a meticulously crafted process of continued pre-training and weight merging. Our approach achieves performance improvements across benchmarks in these languages, exceeding the capabilities of the official Llama-3 models. We provide the model checkpoints as a resource to support further research and development in cross-lingual language understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑