arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.05428cs.CL
收藏

语言模型节食:通过额外预训练高成本效益地开发近缘语言编码器

Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining

Jožef Stefan Institute(约热夫·斯泰凡研究所);University of Ljubljana(卢布尔雅那大学);Masaryk University(马萨里克大学);University of Groningen(格罗宁根大学)

Nikola Ljubešić, Vít Suchomel, Peter Rupnik, Taja Kuzman, Rik van Noord

AI总结:

针对克罗地亚语等近缘语言,研究通过多语言模型额外预训练构建编码器,并在多样化基准上证明其能以有限计算达到接近从头训练模型的性能。

摘要:

语言模型世界正经历动荡时期,性能更强、规模更大的模型正以前所未有的速度涌现。然而,我们认为,尤其对科学界而言,参数量不超过10亿的编码器模型仍然非常必要,其主要用途是利用下游研究所需的元数据来丰富大规模数据集合。我们研究了确保克罗地亚语、塞尔维亚语、波斯尼亚语和黑山语这一组极为近缘语言拥有此类编码器模型的最佳方式:为此类语言建立多样化基准,并将从头训练的模型与通过对现有多语言模型进行额外预训练所构建的新模型进行比较。我们表明,即使计算量有限,通过对可用的多语言模型进行额外预训练,也能获得与专门的从头训练模型相当的性能。我们还表明,邻近语言(在本研究中为斯洛文尼亚语)可以被纳入额外预训练,而最终模型的性能几乎没有损失,甚至完全没有损失。

英文摘要:

The world of language models is going through turbulent times, better and ever larger models are coming out at an unprecedented speed. However, we argue that, especially for the scientific community, encoder models of up to 1 billion parameters are still very much needed, their primary usage being in enriching large collections of data with metadata necessary for downstream research. We investigate the best way to ensure the existence of such encoder models on the set of very closely related languages - Croatian, Serbian, Bosnian and Montenegrin, by setting up a diverse benchmark for these languages, and comparing the trained-from-scratch models with the new models constructed via additional pretraining of existing multilingual models. We show that comparable performance to dedicated from-scratch models can be obtained by additionally pretraining available multilingual models even with a limited amount of computation. We also show that neighboring languages, in our case Slovenian, can be included in the additional pretraining with little to no loss in the performance of the final model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑