arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.02439cs.CLcs.AI
收藏

面向俄语分析的长输入基准

Long Input Benchmark for Russian Analysis

SaluteDevices(萨卢特设备公司);MIPT(莫斯科物理技术学院);AIRI(人工智能研究中心)

Igor Churin, Murat Apishev, Maria Tikhonova, Denis Shevelev, Aydar Bulatov, Yuri Kuratov, Sergej Averkiev, Alena Fenogenova

AI总结:

本文提出俄语长输入基准 LIBRA,包含 21 个改编数据集,分四个复杂度组,用于评估 LLM 在 4k 至 128k token 上下文长度上的长文本理解能力,并提供开源资源。

摘要:

自然语言处理(NLP)的最新进展促进了大型语言模型(LLMs)的发展,这些模型能够解决种类繁多的任务。其应用的一个关键方面是它们处理长文本文档和处理长 token 序列的能力。这催生了对长上下文理解进行恰当评估的需求。为了满足俄语的这一需求,我们提出了 LIBRA(Long Input Benchmark for Russian Analysis,面向俄语分析的长输入基准),它包含 21 个经过改编的数据集,用于深入研究 LLM 彻底理解长文本的能力。这些测试分为四个复杂度组,并允许在从 4k 到 128k token 的各种上下文长度上评估模型。我们为 LIBRA 提供开源数据集、代码库和公开排行榜,以指导未来的研究。

英文摘要:

Recent advancements in Natural Language Processing (NLP) have fostered the development of Large Language Models (LLMs) that can solve an immense variety of tasks. One of the key aspects of their application is their ability to work with long text documents and to process long sequences of tokens. This has created a demand for proper evaluation of long-context understanding. To address this need for the Russian language, we propose LIBRA (Long Input Benchmark for Russian Analysis), which comprises 21 adapted datasets to study the LLM's abilities to understand long texts thoroughly. The tests are divided into four complexity groups and allow the evaluation of models across various context lengths ranging from 4k up to 128k tokens. We provide the open-source datasets, codebase, and public leaderboard for LIBRA to guide forthcoming research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑