arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.25263cs.CLcs.AI
收藏

Mimir:大规模多语言概念建模

Mimir: Large-scale Multilingual Concept Modeling

Department of Computer Science(计算机科学系);University of Bari Aldo Moro(巴里阿尔多·莫罗大学)

Elio Musacchio, Lucia Siciliani, Pierpaolo Basile

AI总结:

提出Mimir,一个1.6B参数的大规模概念模型,通过多语言预训练和指令微调实现概念级别的理解与生成,替代传统的token预测范式。

摘要:

当前的语言建模方法围绕token构建。文本语料被分割成token,模型通过对这些token进行计算来训练,例如根据前文预测下一个token。这一范式已成为现代语言建模的标准,尤其是基于token的架构取得了卓越性能。然而,最近的研究不仅开始质疑语言模型如何从token中处理和理解意义,还开始质疑使用更高级别的粒度是否能推动研究领域的发展。这引出了概念建模的想法,即直接训练模型进行下一个概念预测,而非下一个token预测。目标是输入从token转变为概念,迫使底层语言模型将其粒度从细粒度的token转变为广泛的概念。在这项工作中,我们介绍了Mimir,一个1.6B参数的大规模概念模型,用于多语言概念理解和生成。我们利用了一个大规模多语言预训练语料库(38,883,987,240个句子),涵盖46种语言,以及一个大规模多轮多语言指令微调数据集(66,816,428个句子),覆盖总共35种语言。我们针对一个参数数量相当的语言模型,对模型性能进行了广泛评估。

英文摘要:

Current language modeling approaches are built around tokens. Text corpora are split into tokens, and models are trained by performing computations on these tokens, such as predicting the next token given the preceding ones as context. This paradigm has become the standard in modern language modeling, especially given the outstanding performance obtained by token-based architectures. However, recent works have not only begun to question how language models process and understand meaning from tokens, but also to question whether using higher levels of granularity could advance the research field. This led to the idea of Concept Modeling, that is, to directly train models for next-concept prediction rather than next-token prediction. In this work, we introduce Mimir, a 1.6B Large Concept Model trained for multilingual concept understanding and generation. We leverage a large-scale multilingual pre-training corpus (38,883,987,240 sentences) spanning 46 languages and a large-scale multi-turn, and multilingual instruction-tuning dataset (66,816,428 sentences) covering a total of 35 languages. We extensively evaluate model performance against a language model with a comparable number of parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑