arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.05975cs.CLcs.AI
收藏

LLaMAX:通过增强超越100种语言的翻译能力来扩展LLM的语言视野

LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages

Shanghai AI Laboratory(上海人工智能实验室);Nanjing University(南京大学);Carnegie Mellon University(卡内基梅隆大学)

Yinquan Lu, Wenhao Zhu, Lei Li, Yu Qiao, Fei Yuan

AI总结:

本文通过增强LLaMA系列模型的多语言持续预训练,提出LLaMAX模型,支持100多种语言翻译,在不损失泛化能力下显著超越现有开源LLM并媲美专用翻译模型。

摘要:

大型语言模型(LLMs)在高资源语言任务中展现出卓越的翻译能力,但由于预训练期间多语言数据不足,其在低资源语言上的表现受到阻碍。为解决此问题,我们对LLaMA系列模型进行了广泛的多语言持续预训练,实现了对100多种语言的翻译支持。通过对词汇扩展和数据增强等训练策略的全面分析,我们开发了LLaMAX。值得注意的是,在不牺牲泛化能力的情况下,与现有的开源LLMs相比,LLaMAX实现了显著更高的翻译性能(高出10个spBLEU点以上),并在Flores-101基准上与专门的翻译模型(M2M-100-12B)表现相当。大量实验表明,LLaMAX可作为强大的多语言基础模型。代码和模型已公开发布。

英文摘要:

Large Language Models (LLMs) demonstrate remarkable translation capabilities in high-resource language tasks, yet their performance in low-resource languages is hindered by insufficient multilingual data during pre-training. To address this, we conduct extensive multilingual continual pre-training on the LLaMA series models, enabling translation support across more than 100 languages. Through a comprehensive analysis of training strategies, such as vocabulary expansion and data augmentation, we develop LLaMAX. Remarkably, without sacrificing its generalization ability, LLaMAX achieves significantly higher translation performance compared to existing open-source LLMs (by more than 10 spBLEU points) and performs on-par with specialized translation model (M2M-100-12B) on the Flores-101 benchmark. Extensive experiments indicate that LLaMAX can serve as a robust multilingual foundation model. The code \footnote{\url{https://github.com/CONE-MT/LLaMAX/.}} and the models \footnote{\url{https://huggingface.co/LLaMAX/.}} are publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑