arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.00786cs.CLcs.LG
收藏

CroissantLLM:一个真正双语的法语-英语语言模型

CroissantLLM: A Truly Bilingual French-English Language Model

Illuin Technology(伊卢因科技);Unbabel(昂巴贝尔公司);Equall(伊阔尔公司);Diabolocom(迪亚博洛科姆公司);MICS, CentraleSupélec(中央理工学院MICS实验室);Instituto de Telecomunicações, Lisboa(里斯本电信研究所);INESC-ID, Lisboa(里斯本INESC-ID研究所);Instituto Superior Técnico, Universidade de Lisboa(里斯本大学高等技术学院);Sorbonne Université, CNRS, ISIR, Paris(巴黎索邦大学法国国家科学研究中心智能与机器人系统研究所);Imperial College London(伦敦帝国学院);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)
Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, An… 展开作者

Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, Pierre Colombo

AI总结:

CroissantLLM是一个在3T英法token上以1:1比例预训练的1.3B开源双语模型,通过自定义分词器、双语微调数据集及FrenchBench基准,在消费级硬件上实现高性能法语-英语能力并验证81%透明度标准。

摘要:

我们介绍了CroissantLLM,一个在3T英语和法语token集合上预训练的1.3B语言模型,旨在为研究和工业界带来一个高性能、完全开源的双语模型,能够在消费级本地硬件上快速运行。为此,我们开创了一种训练内在双语模型的方法,采用1:1的英语与法语预训练数据比例、自定义分词器和双语微调数据集。我们发布了训练数据集,其中特别包含一个法语部分,具有人工整理、高质量且多样化的数据来源。为了评估英语之外的表现,我们构建了一个新颖的基准FrenchBench,由一系列分类和生成任务组成,涵盖法语中模型性能的各个正交方面。此外,基于透明性原则并为了促进进一步的大语言模型研究,我们发布了代码库、跨不同模型规模、训练数据分布和训练步骤的数十个检查点,以及微调后的Chat模型和强大的翻译模型。我们通过FMTI框架评估我们的模型,并验证了81%的透明度标准,远超甚至大多数开放计划的得分。这项工作丰富了NLP领域,打破了以往以英语为中心的工作,以加强我们对语言模型中多语言性的理解。

英文摘要:

We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local hardware. To that end, we pioneer the approach of training an intrinsically bilingual model with a 1:1 English-to-French pretraining data ratio, a custom tokenizer, and bilingual finetuning datasets. We release the training dataset, notably containing a French split with manually curated, high-quality, and varied data sources. To assess performance outside of English, we craft a novel benchmark, FrenchBench, consisting of an array of classification and generation tasks, covering various orthogonal aspects of model performance in the French Language. Additionally, rooted in transparency and to foster further Large Language Model research, we release codebases, and dozens of checkpoints across various model sizes, training data distributions, and training steps, as well as fine-tuned Chat models, and strong translation models. We evaluate our model through the FMTI framework, and validate 81 % of the transparency criteria, far beyond the scores of even most open initiatives. This work enriches the NLP landscape, breaking away from previous English-centric work in order to strengthen our understanding of multilinguality in language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑