arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.09296cs.CL
收藏

KoLA:严谨评测大语言模型的世界知识

KoLA: Carefully Benchmarking World Knowledge of Large Language Models

Tsinghua University(清华大学)
Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, Chunyang Li, Zheyuan Zhang, Yushi Bai… 展开作者

Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, Chunyang Li, Zheyuan Zhang, Yushi Bai, Yantao Liu, Amy Xin, Nianyi Lin, Kaifeng Yun, Linlu Gong, Jianhui Chen, Zhili Wu, Yunjia Qi, Weikai Li, Yong Guan, Kaisheng Zeng, Ji Qi, Hailong Jin, Jinxin Liu, Yu Gu, Yuan Yao, Ning Ding, Lei Hou, Zhiyuan Liu, Bin Xu, Jie Tang, Juanzi Li

AI总结:

本文提出知识导向的大语言模型评测基准 KoLA,通过四层能力分类、静态与持续更新的数据以及对比式评测标准,系统评估 28 个模型的世界知识能力。

摘要:

大语言模型(LLMs)前所未有的性能表现迫切要求评测方法加以改进。我们认为,不应仅仅探索 LLM 能力的广度,细致而深思熟虑的设计对于实现全面、无偏且具有应用价值的评测至关重要。鉴于世界知识对 LLM 的重要性,我们构建了一个面向知识的 LLM 评估基准(KoLA),并在其中精心设计了三个关键因素:(1)在能力建模方面,我们模仿人类认知,构建了知识相关能力的四层分类体系,涵盖 19 个任务。(2)在数据方面,为确保公平比较,我们同时使用 LLM 普遍预训练过的语料库 Wikipedia,以及持续收集的新兴语料,旨在评估模型处理未见数据和不断演化知识的能力。(3)在评测标准方面,我们采用对比式体系,包括用于提升跨任务和跨模型数值可比性的总体标准分数,以及用于自动评估知识创造能力的独特自对比指标。我们评测了 28 个开源和商用 LLM,并获得了一些有趣的发现。KoLA 数据集和开放参与的排行榜已在 https://kola.xlore.cn 公开发布,并将持续更新,为开发 LLM 和知识相关系统提供参考。

英文摘要:

The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations. Rather than merely exploring the breadth of LLM abilities, we believe meticulous and thoughtful designs are essential to thorough, unbiased, and applicable evaluations. Given the importance of world knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark (KoLA), in which we carefully design three crucial factors: (1) For \textbf{ability modeling}, we mimic human cognition to form a four-level taxonomy of knowledge-related abilities, covering $19$ tasks. (2) For \textbf{data}, to ensure fair comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously collected emerging corpora, aiming to evaluate the capacity to handle unseen data and evolving knowledge. (3) For \textbf{evaluation criteria}, we adopt a contrastive system, including overall standard scores for better numerical comparability across tasks and models and a unique self-contrast metric for automatically evaluating knowledge-creating ability. We evaluate $28$ open-source and commercial LLMs and obtain some intriguing findings. The KoLA dataset and open-participation leaderboard are publicly released at https://kola.xlore.cn and will be continuously updated to provide references for developing LLMs and knowledge-related systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑