NeoDictaBERT：推动希伯来语BERT模型的前沿发展
NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew
AI总结：
本文提出NeoDictaBERT系列模型，采用NeoBERT架构专门训练希伯来语，在希伯来语基准上超越现有模型，其中双语版在检索任务上表现突出，并已开源。
摘要：
自首次发布以来，BERT模型尽管规模相对较小（BERT-base约有1亿个参数），但在各种任务上表现出卓越的性能。然而，与Llama3和Qwen3等更新的基于Transformer的模型相比，这些模型所使用的架构选择已经过时。近几个月来，已有几种架构被提出以缩小这一差距。ModernBERT和NeoBERT均在英语基准测试中展现出显著的改进，并大幅扩展了支持的上下文窗口。继它们的成功之后，我们推出了NeoDictaBERT和NeoDictaBERT-bilingual：采用与NeoBERT相同架构训练的BERT风格模型，专门聚焦于希伯来语文本。这些模型在几乎所有希伯来语基准测试上均优于现有模型，并为下游任务提供了坚实的基础。值得注意的是，NeoDictaBERT-bilingual模型在检索任务上表现出强劲的结果，超越了其他类似规模的多语言模型。在本文中，我们描述了训练过程，并报告了在各种基准测试上的结果。我们将这些模型发布给社区，作为我们推进希伯来语自然语言处理研究与开发目标的一部分。
英文摘要：
Since their initial release, BERT models have demonstrated exceptional performance on a variety of tasks, despite their relatively small size (BERT-base has ~100M parameters). Nevertheless, the architectural choices used in these models are outdated compared to newer transformer-based models such as Llama3 and Qwen3. In recent months, several architectures have been proposed to close this gap. ModernBERT and NeoBERT both show strong improvements on English benchmarks and significantly extend the supported context window. Following their successes, we introduce NeoDictaBERT and NeoDictaBERT-bilingual: BERT-style models trained using the same architecture as NeoBERT, with a dedicated focus on Hebrew texts. These models outperform existing ones on almost all Hebrew benchmarks and provide a strong foundation for downstream tasks. Notably, the NeoDictaBERT-bilingual model shows strong results on retrieval tasks, outperforming other multilingual models of similar size. In this paper, we describe the training process and report results across various benchmarks. We release the models to the community as part of our goal to advance research and development in Hebrew NLP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
