arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.12191cs.CL
收藏

长上下文编码器模型用于波兰语言理解

Long-Context Encoder Models for Polish Language Understanding

National Information Processing Institute(国家信息处理研究所);PKO Bank Polski(波兰PKO银行)

Sławomir Dadas, Rafał Poświata, Marek Kozłowski, Małgorzata Grębowiec, Michał Perełkiewicz, Paweł Klimiuk, Przemysław Boruta

AI总结:

本文提出了一种能够处理长文档的波兰语言模型,通过两阶段训练和知识蒸馏压缩模型,在25个任务中展现优异的长上下文处理能力。

摘要:

尽管解码器-only的大型语言模型(LLMs)最近在NLP领域占据主导地位,但仅编码器的架构仍然是判别任务中成本效益高且参数效率高的标准。然而,经典编码器如BERT受限于短上下文窗口,无法处理长文档。在本文中,我们针对波兰语言引入了一个高质量的波兰模型,能够处理最多8192个标记的序列。该模型通过采用两阶段训练程序,包括位置嵌入适应和全参数连续预训练来开发。此外,我们还提出了通过知识蒸馏训练的压缩模型变体。这些模型在25个任务上进行了评估,包括KLEJ基准、新引入的金融任务集(FinBench)以及其他分类和回归任务,特别是需要长文档理解的任务。结果表明,我们的模型在波兰语和多语言模型中实现了最佳的平均性能,在长上下文任务中显著优于竞争解决方案,同时在短文本上保持了可比的质量。

英文摘要:

While decoder-only Large Language Models (LLMs) have recently dominated the NLP landscape, encoder-only architectures remain a cost-effective and parameter-efficient standard for discriminative tasks. However, classic encoders like BERT are limited by a short context window, which is insufficient for processing long documents. In this paper, we address this limitation for the Polish by introducing a high-quality Polish model capable of processing sequences of up to 8192 tokens. The model was developed by employing a two-stage training procedure that involves positional embedding adaptation and full parameter continuous pre-training. Furthermore, we propose compressed model variants trained via knowledge distillation. The models were evaluated on 25 tasks, including the KLEJ benchmark, a newly introduced financial task suite (FinBench), and other classification and regression tasks, specifically those requiring long-document understanding. The results demonstrate that our model achieves the best average performance among Polish and multilingual models, significantly outperforming competitive solutions in long-context tasks while maintaining comparable quality on short texts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑