arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.15147cs.CL
收藏

S3Eval:一个面向大语言模型的合成、可扩展、系统化评估套件

S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models

Institute of Automation, CAS(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Sea AI Lab(海洋人工智能实验室)

Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, Kang Liu

AI总结:

针对超长上下文能力难以人工评估的问题,本文提出可控制长度与难度的合成评估套件 S3Eval,并以 S3Eval-Standard 验证其与真实基准相关且对现有 LLM 具有挑战性。

摘要:

大语言模型(LLM)的快速发展,推动长上下文理解和推理等模型能力取得巨大进步。然而,随着 LLM 能够处理更长的上下文,评估它们是否已经获得某些能力变得更具挑战性,因为其可处理的文本长度(例如 200K tokens)远远超出人类能够在合理时间内可靠评估的范围。本文提出使用复杂合成任务作为一种代理评估方法,并介绍 S3Eval,这是一个用于 LLM 评估的合成、可扩展、系统化评估套件。S3Eval 的合成特性使用户能够完全控制数据集,从而通过在多种场景中扩展文本长度并改变任务难度,系统化地探测 LLM 能力。S3Eval 与真实世界基准之间的强相关性证明了使用 S3Eval 评估 LLM 的合理性。S3Eval 提供灵活且无限的长上下文数据生成方法。我们已经生成了一个名为 S3Eval-Standard 的综合数据集,实验结果表明,它对所有现有 LLM 都构成了重大挑战。

英文摘要:

The rapid development of Large Language Models (LLMs) has led to great strides in model capabilities like long-context understanding and reasoning. However, as LLMs are able to process longer contexts, it becomes more challenging to evaluate whether they have acquired certain capabilities, since the length of text (e.g., 200K tokens) they can process far exceeds what humans can reliably assess in a reasonable duration. In this paper, we propose using complex synthetic tasks as a proxy evaluation method, and present S3Eval, a Synthetic, Scalable, Systematic evaluation suite for LLMs evaluation. The synthetic nature of S3Eval provides users full control over the dataset, allowing them to systematically probe LLM capabilities by scaling text length and varying task difficulty across diverse scenarios. The strong correlation between S3Eval and real-world benchmarks demonstrates the soundness of using S3Eval for evaluation of LLMs. S3Eval provides a flexible and infinite long-context data generation method. We have generated a comprehensive dataset called S3Eval-Standard, and experimental results have shown that it poses significant challenges for all existing LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑