CMMLU：衡量中文的大规模多任务语言理解能力
CMMLU: Measuring massive multitask language understanding in Chinese
AI总结：
本文提出涵盖多学科的中文基准 CMMLU，系统评估 18 个先进 LLM，发现多数模型平均准确率不足 50%，揭示中文知识与推理能力的改进方向。
摘要：
随着大型语言模型（LLM）的能力不断进步，评估其性能变得愈发关键且富有挑战性。本文旨在通过引入 CMMLU 来弥合这一差距，这是一个综合性中文基准，涵盖自然科学、社会科学、工程和人文学科等各类学科。我们对 18 个先进的面向多语言和中文的 LLM 进行了全面评估，考查它们在不同学科和设置下的性能。结果表明，即使提供了上下文示例和思维链提示，大多数现有 LLM 仍难以达到 50% 的平均准确率，而随机基线为 25%。这凸显出 LLM 仍有显著的改进空间。此外，我们开展了大量实验以识别影响模型性能的因素，并提出增强 LLM 的方向。CMMLU 填补了在中文语境下评估大型语言模型知识与推理能力方面的空白。
英文摘要：
As the capabilities of large language models (LLMs) continue to advance, evaluating their performance becomes increasingly crucial and challenging. This paper aims to bridge this gap by introducing CMMLU, a comprehensive Chinese benchmark that covers various subjects, including natural science, social sciences, engineering, and humanities. We conduct a thorough evaluation of 18 advanced multilingual- and Chinese-oriented LLMs, assessing their performance across different subjects and settings. The results reveal that most existing LLMs struggle to achieve an average accuracy of 50%, even when provided with in-context examples and chain-of-thought prompts, whereas the random baseline stands at 25%. This highlights significant room for improvement in LLMs. Additionally, we conduct extensive experiments to identify factors impacting the models' performance and propose directions for enhancing LLMs. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models within the Chinese context.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
