arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21379cs.CLcs.AIcs.LG
收藏

MrBERT:通过词汇、领域和维度适应构建的现代多语言编码器

MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation

Barcelona Supercomputing Center(巴塞罗那超级计算中心)

Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo, Severino Da Dalt, Javier Aula-Blasco, Aitor Gonzalez-Agirre, Marta Villegas

AI总结:

MrBERT通过词汇、领域和维度适应,实现了多语言编码器在特定任务和专业领域中的高性能与高效优化。

摘要:

我们介绍了MrBERT,一种基于ModernBERT架构、参数在1.5亿至3亿之间的编码器家族,其在35种语言和代码上进行预训练。通过针对性适应,该模型家族在加泰罗尼亚语和西班牙语特定任务上实现了最先进的结果,同时在专门的生物医学和法律领域建立了稳健的性能。为弥合研究与生产的差距,我们集成了Matryoshka Representation Learning(MRL),使向量大小具有灵活性,显著降低了推理和存储成本。最终,MrBERT家族证明了现代编码器架构可以同时优化局部语言卓越性和高效、高风险领域的专业化。我们已在Huggingface上开源完整的模型家族。

英文摘要:

We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains. To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs. Ultimately, the MrBERT family demonstrates that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization. We open source the complete model family on Huggingface.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑