RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
Sberbank(俄罗斯联邦储蓄银行);National Research University Higher School of Economics(国立研究型大学高等经济学院);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Huawei Noah’s Ark lab(华为诺亚方舟实验室)
Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, Andrey Evlampiev
In this paper, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE. Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language. We provide baselines, human level evaluation, an open-source framework for evaluating models (https://github.com/RussianNLP/RussianSuperGLUE), and an overall leaderboard of transformer models for the Russian language. Besides, we present the first results of comparing multilingual models in the adapted diagnostic test set and offer the first steps to further expanding or assessing state-of-the-art models independently of language.
Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models
SberDevices;Sberbank(俄罗斯联邦储蓄银行);National Research University Higher School of Economics(国立研究型高等经济学院);ANO «AI Research Institute»(“人工智能研究院”自治非营利组织);Alex Kukushkin Lab(Alex Kukushkin实验室);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Huawei Noah’s Ark lab(华为诺亚方舟实验室);Kazan Federal University(喀山联邦大学)
CRAFT: Cultural Russian-Oriented Dataset Adaptation for Focused Text-to-Image Generation
Sber AI;Moscow Institute of Physics and Technology (MIPT)(莫斯科物理技术学院);Information Technologies, Mechanics and Optics University (ITMO University)(圣彼得堡国立信息技术、机械与光学大学);Artificial Intelligence Research Institute (AIRI)(人工智能研究所)
RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs
University of Edinburgh(爱丁堡大学);HSE University(高等经济大学);University of Groningen(格罗宁根大学);Ghent University(根特大学);SaluteDevices(SaluteDevices公司);Toloka AI(Toloka AI公司);University of Oslo(奥斯陆大学)
A Culturally-diverse Multilingual Multimodal Video Benchmark & Model
Mohamed bin Zayed University of Artificial Intelligence(莫德赫·本·扎耶德人工智能大学);University of Central Florida(中央佛罗里达大学);Islamic University of Technology(伊斯兰技术大学);Air University(空军大学);ETH Zurich(苏黎世联邦理工学院);Technische Universität München(慕尼黑技术大学);National Institute of Informatics(国家信息研究所);Australian National University(澳大利亚国立大学);Linköping University(利尔贝里大学)