Over the past few years, one of the most notable advancements in AI research has been in foundation models (FMs), headlined by the rise of language models (LMs). As the models' size increases, LMs demonstrate enhancements in measurable aspects and the development of new qualitative features. However, despite researchers' attention and the rapid growth in LM application, the capabilities, limitations, and associated risks still need to be better understood. To address these issues, we introduce an open Multimodal Evaluation of Russian-language Architectures (MERA), a new instruction benchmark for evaluating foundation models oriented towards the Russian language. The benchmark encompasses 21 evaluation tasks for generative models in 11 skill domains and is designed as a black-box test to ensure the exclusion of data leakage. The paper introduces a methodology to evaluate FMs and LMs in zero- and few-shot fixed instruction settings that can be extended to other modalities. We propose an evaluation methodology, an open-source code base for the MERA assessment, and a leaderboard with a submission system. We evaluate open LMs as baselines and find that they are still far behind the human level. We publicly release MERA to guide forthcoming research, anticipate groundbreaking model features, standardize the evaluation procedure, and address potential societal drawbacks.
Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels
Zhejiang University(浙江大学);School of Engineering, Westlake University(西湖大学工程学院);University College London(伦敦大学学院);University of Cambridge(剑桥大学);LanBridge Group(兰桥集团);Institute of Advanced Technology, Westlake Institute for Advanced Study(西湖高等研究院先进技术研究院)
AGRR-2019: A Corpus for Gapping Resolution in Russian
ABBYY(ABBYY公司);Charles University(查理大学);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Sberbank(俄罗斯联邦储蓄银行);National Research University Higher School of Economics(国家研究型高等经济大学)
PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Zhejiang University(浙江大学);CAS Key Laboratory of AI Security, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);Fudan University(复旦大学)
Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxification
AIRI;Universidade de Santiago de Compostela(圣塞巴斯蒂安大学);Technical University of Munich (TUM)(慕尼黑技术大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Skoltech
XTRUST: On the Multilingual Trustworthiness of Large Language Models
School of Artificial Intelligence, Jilin University, China(吉林大学人工智能学院);The Hong Kong University of Science and Technology (Guangzhou), China(香港科学与技术大学(广州));Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University, China(未来科学国际中心)