大型语言模型能否理解古汉语？ACLUE 的初步测试
Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE
AI总结：
本文提出ACLUE基准，含15项古汉语理解任务，评估8个LLM，发现其现代与古汉语表现差异大，ChatGLM2最佳，平均37.4%，并公开代码数据。
摘要：
大型语言模型（LLMs）在理解和生成语言方面展现了卓越的能力。然而，它们在理解古代语言，尤其是古汉语方面的能力在很大程度上仍未得到探索。为了弥补这一空白，我们提出了ACLUE，一个旨在评估语言模型理解古汉语能力的评测基准。ACLUE包含15项任务，涵盖语音、词汇、句法、语义、推理和知识等一系列技能。通过对八个最先进的大型语言模型的评估，我们观察到它们在现代汉语和古汉语之间的表现存在显著差异。在评估的模型中，ChatGLM2表现最为突出，平均得分达到37.4%。我们已公开提供我们的代码和数据。
英文摘要：
Large language models (LLMs) have showcased remarkable capabilities in understanding and generating language. However, their ability in comprehending ancient languages, particularly ancient Chinese, remains largely unexplored. To bridge this gap, we present ACLUE, an evaluation benchmark designed to assess the capability of language models in comprehending ancient Chinese. ACLUE consists of 15 tasks cover a range of skills, spanning phonetic, lexical, syntactic, semantic, inference and knowledge. Through the evaluation of eight state-of-the-art LLMs, we observed a noticeable disparity in their performance between modern Chinese and ancient Chinese. Among the assessed models, ChatGLM2 demonstrates the most remarkable performance, achieving an average score of 37.4%. We have made our code and data public available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
