罗马尼亚BERT的诞生
The birth of Romanian BERT
摘要：
大规模预训练语言模型在自然语言处理中已经变得无处不在。然而，这些模型中的大多数要么以高资源语言（特别是英语）提供，要么作为多语言模型提供，这些模型会牺牲单个语言的性能。本文介绍了罗马尼亚语BERT，第一个纯粹的罗马尼亚语转换为基础的语言模型，预训练的大型文本语料库。我们讨论了语料库的组成和清洗，模型的训练过程，以及广泛的评估模型上的各种罗马尼亚数据集。我们不仅开源了模型本身，还开源了一个存储库，其中包含如何获取语料库，在生产中微调和使用此模型（使用实际示例）以及如何完全复制评估过程的信息。
英文摘要：
Large-scale pretrained language models have become ubiquitous in Natural Language Processing. However, most of these models are available either in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. This paper introduces Romanian BERT, the first purely Romanian transformer-based language model, pretrained on a large text corpus. We discuss corpus composition and cleaning, the model training process, as well as an extensive evaluation of the model on various Romanian datasets. We open source not only the model itself, but also a repository that contains information on how to obtain the corpus, fine-tune and use this model in production (with practical examples), and how to fully replicate the evaluation process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
