arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.00630cs.CL
收藏

KLEJ:波兰语语言理解的综合基准

KLEJ: Comprehensive Benchmark for Polish Language Understanding

AGH University of Science and Technology(AGH科技大学)

Piotr Rybak, Robert Mroczkowski, Janusz Tracz, Ireneusz Gawlik

摘要:

近年来,一系列基于Transformer的模型在一般自然语言理解(NLU)任务中取得了重大改进。如果没有一般的NLU基准,这样快的研究速度是不可能的,这些基准允许对所提出的方法进行公平的比较。然而,这样的基准只适用于少数几种语言。为了缓解这个问题,我们引入了一个全面的多任务基准波兰语语言理解,伴随着在线排行榜。它由一组不同的任务组成,这些任务来自现有的数据集,用于命名实体识别、问答、文本蕴涵等。我们还介绍了一个新的情感分析任务的电子商务领域,命名为快板评论(AR)。为了确保一个通用的评估方案,并促进模型推广到不同的NLU任务,基准测试包括来自不同领域和应用程序的数据集。此外,我们还发布了HerBERT,这是一个专门针对波兰语训练的基于transformer的模型,它具有最佳的平均性能,并在九个任务中的三个任务中获得了最佳结果。最后,我们提供了一个广泛的评估,包括几个标准基线和最近提出的基于多语言transformer的模型。

英文摘要:

In recent years, a series of Transformer-based models unlocked major improvements in general natural language understanding (NLU) tasks. Such a fast pace of research would not be possible without general NLU benchmarks, which allow for a fair comparison of the proposed methods. However, such benchmarks are available only for a handful of languages. To alleviate this issue, we introduce a comprehensive multi-task benchmark for the Polish language understanding, accompanied by an online leaderboard. It consists of a diverse set of tasks, adopted from existing datasets for named entity recognition, question-answering, textual entailment, and others. We also introduce a new sentiment analysis task for the e-commerce domain, named Allegro Reviews (AR). To ensure a common evaluation scheme and promote models that generalize to different NLU tasks, the benchmark includes datasets from varying domains and applications. Additionally, we release HerBERT, a Transformer-based model trained specifically for the Polish language, which has the best average performance and obtains the best results for three out of nine tasks. Finally, we provide an extensive evaluation, including several standard baselines and recently proposed, multilingual Transformer-based models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑