arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.04531cs.CLcs.AI
收藏

MERA:俄语大语言模型综合评测基准

MERA: A Comprehensive LLM Evaluation in Russian

SaluteDevices;HSE University(高等经济大学);Center for Artificial Intelligence Technology(人工智能技术中心);AIRI
Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, L… 展开作者

Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Ulyana Isaeva, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Denis Dimitrov, Alexander Panchenko, Sergei Markov

AI总结:

本文提出俄语基础模型评测基准MERA,涵盖11个技能领域21项黑盒任务,并公开代码库与排行榜,评测表明现有开放语言模型仍远低于人类水平。

摘要:

过去几年中,人工智能研究最显著的进展之一是以语言模型(LMs)为代表的基础模型(FMs)的兴起。随着模型规模的增大,语言模型在可量化方面表现出提升,并发展出新的定性特征。然而,尽管研究者给予了大量关注且语言模型应用快速增长,其能力、局限性和相关风险仍需要更深入的理解。为解决这些问题,我们推出了一个开放的俄语架构多模态评测基准(MERA),这是一个面向俄语基础模型的新型指令评测基准。该基准涵盖生成式模型在11个技能领域中的21项评测任务,并设计为黑盒测试以确保排除数据泄漏。本文介绍了一种在零样本和少样本固定指令设置下评测基础模型和语言模型的方法,该方法可扩展到其他模态。我们提出了评测方法、用于MERA评估的开源代码库,以及带有提交系统的排行榜。我们对开放语言模型进行了基线评测,发现它们仍远未达到人类水平。我们公开发布MERA,以指导未来研究、预见突破性模型特性、标准化评测流程,并应对潜在的社会性缺陷。

英文摘要:

Over the past few years, one of the most notable advancements in AI research has been in foundation models (FMs), headlined by the rise of language models (LMs). As the models' size increases, LMs demonstrate enhancements in measurable aspects and the development of new qualitative features. However, despite researchers' attention and the rapid growth in LM application, the capabilities, limitations, and associated risks still need to be better understood. To address these issues, we introduce an open Multimodal Evaluation of Russian-language Architectures (MERA), a new instruction benchmark for evaluating foundation models oriented towards the Russian language. The benchmark encompasses 21 evaluation tasks for generative models in 11 skill domains and is designed as a black-box test to ensure the exclusion of data leakage. The paper introduces a methodology to evaluate FMs and LMs in zero- and few-shot fixed instruction settings that can be extended to other modalities. We propose an evaluation methodology, an open-source code base for the MERA assessment, and a leaderboard with a submission system. We evaluate open LMs as baselines and find that they are still far behind the human level. We publicly release MERA to guide forthcoming research, anticipate groundbreaking model features, standardize the evaluation procedure, and address potential societal drawbacks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑