FoundaBench：评估大型语言模型的中国基础知识能力
FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models
AI总结：
本文提出FoundaBench基准，包含3354道多项选择题以评估中文LLMs的基础知识能力，通过广泛评估12个模型发现中文语料库预训练模型表现更优，并揭示了推理与记忆能力的差异。
摘要：
在蓬勃发展的大型语言模型（LLMs）领域中，基础知识的评估仍然是一个关键挑战，特别是对于针对中文语言和文化定制的模型而言。本文介绍了FoundaBench，这是一个开创性的基准，旨在严格评估中文LLMs的基础知识能力。FoundaBench包含涵盖常识和K-12教育学科的3354道多项选择题，经过精心策划以反映日常和学术知识的广度和深度。我们使用FoundaBench对12个最先进的LLMs进行了广泛评估，采用传统评估方法和我们的CircularEval协议来减轻模型响应中的潜在偏差。我们的结果突出了在中文语料库上预训练的模型的卓越性能，并揭示了模型推理与记忆回忆能力之间的显著差异。从FoundaBench评估中获得的见解为理解LLMs的基础知识设定了新标准，为该领域的未来进展提供了稳健的框架。
英文摘要：
In the burgeoning field of large language models (LLMs), the assessment of fundamental knowledge remains a critical challenge, particularly for models tailored to Chinese language and culture. This paper introduces FoundaBench, a pioneering benchmark designed to rigorously evaluate the fundamental knowledge capabilities of Chinese LLMs. FoundaBench encompasses a diverse array of 3354 multiple-choice questions across common sense and K-12 educational subjects, meticulously curated to reflect the breadth and depth of everyday and academic knowledge. We present an extensive evaluation of 12 state-of-the-art LLMs using FoundaBench, employing both traditional assessment methods and our CircularEval protocol to mitigate potential biases in model responses. Our results highlight the superior performance of models pre-trained on Chinese corpora, and reveal a significant disparity between models' reasoning and memory recall capabilities. The insights gleaned from FoundaBench evaluations set a new standard for understanding the fundamental knowledge of LLMs, providing a robust framework for future advancements in the field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
