arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.15925cs.CLcs.AI
收藏

RussianSuperGLUE:一个俄语语言理解评估基准

RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark

Sberbank(俄罗斯联邦储蓄银行);National Research University Higher School of Economics(国立研究型大学高等经济学院);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Huawei Noah’s Ark lab(华为诺亚方舟实验室)

Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, Andrey Evlampiev

AI总结:

本文推出俄语版 SuperGLUE 基准 RussianGLUE,含九项任务、基线、人类评估、开源框架与排行榜,并比较多语言模型。

摘要:

在本文中,我们介绍了一个先进的俄语通用语言理解评估基准——RussianGLUE。通用语言模型和 transformer 领域的最新进展,要求开发一种对其进行广泛诊断和测试通用智能技能的方法,包括自然语言推理检测、常识推理,以及不受文本主题或词汇影响而执行简单逻辑操作的能力。我们首次从零开始为俄语开发了一个包含九项任务的基准,其收集和组织方式仿照 SuperGLUE 方法。我们提供了基线、人类水平评估、一个用于评估模型的开源框架(https://github.com/RussianNLP/RussianSuperGLUE),以及俄语 transformer 模型的总排行榜。此外,我们还展示了在适配后的诊断测试集上比较多语言模型的首批结果,并为进一步扩展或独立于语言评估最先进模型提供了初步步骤。

英文摘要:

In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE. Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language. We provide baselines, human level evaluation, an open-source framework for evaluating models (https://github.com/RussianNLP/RussianSuperGLUE), and an overall leaderboard of transformer models for the Russian language. Besides, we present the first results of comparing multilingual models in the adapted diagnostic test set and offer the first steps to further expanding or assessing state-of-the-art models independently of language.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑