Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages
University of Ljubljana, Faculty of Computer and Information Science(卢布尔雅那大学计算机与信息科学学院);University of Ljubljana, Faculty of Arts(卢布尔雅那大学艺术学院);Jožef Stefan Institute(乔塞夫·斯塔芬研究所)
Tjaša Arčon, Matej Klemen, Marko Robnik-Šikonja, Kaja Dobrovoljc
LLMs are routinely evaluated on language use, yet their explicit knowledge about linguistic structure remains poorly understood. Existing linguistic benchmarks focus on narrow phenomena, emphasize high-resource languages, and rarely test metalinguistic knowledge - explicit reasoning about language structure. We present a multilingual evaluation of metalinguistic knowledge in LLMs, based on the World Atlas of Language Structures (WALS), documenting 192 linguistic features across 2,660 languages. We convert WALS features into natural-language multiple-choice questions and evaluate models across documented languages. Using accuracy and macro F1, and comparing to chance and majority-class baselines, we assess performance and analyse variation across linguistic domains and language-related factors. Results show limited metalinguistic knowledge: GPT-4o performs best but achieves moderate accuracy (0.367), while open-source models lag. Although all models perform above chance, they fail to outperform the majority-class baseline, suggesting they capture broad cross-linguistic patterns but lack fine-grained distinctions. Performance varies by domain, partly reflecting differences in online visibility. At the language level, accuracy correlates with digital language status: languages with greater digital presence and resources are evaluated more accurately, while low-resource languages perform worse. Analysis of predictive factors confirms that resource-related indicators (Wikipedia size, corpus availability) are more informative than geographic, genealogical, or sociolinguistic factors. Overall, LLM metalinguistic knowledge appears fragmented and shaped mainly by data availability, rather than broadly generalizable grammatical competence. We release the benchmark as an open-source dataset to support evaluation across languages and encourage greater global linguistic diversity in future LLMs.
University of Science and Technology of China(中国科学技术大学);School of Computer Science, Fudan University(复旦大学计算机科学学院);Singapore Management University(新加坡管理大学);Beijing Institute of Technology(北京理工大学)
KMIR: A Benchmark for Evaluating Knowledge Memorization, Identification and Reasoning Abilities of Language Models
Huawei Poisson Laboratory(华为泊松实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Hong Kong University of Science and Technology(香港科技大学)
SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University(浙江大学Hangzhou全球科技创新中心);ZJU-UIUC Institute, Zhejiang University(浙江大学UIUC研究院)
OpenLearnLM Benchmark: A Unified Framework for Evaluating Knowledge, Skill, and Attitude in Educational Large Language Models
Chosun University(chosun大学);Korea University(韩国大学);Ewha Womans University(成均馆大学);Korea Institute for Curriculum and Evaluation(韩国课程评价院);Seoul National University(首尔国立大学);Texas A&M University(德克萨斯农工大学);Indiana University Bloomington(印第安纳大学布卢明顿分校)
GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning
MemTensor (Shanghai) Technology Co., Ltd.(MemTensor(上海)科技有限公司);University of Sydney(悉尼大学);Research Institute of China Telecom(中国电信研究院);Renmin University of China(中国人民大学)
Bridging the Knowledge-Action Gap by Evaluating LLMs in Dynamic Dental Clinical Scenarios
The Second Affiliated Hospital of Harbin Medical University(哈尔滨医科大学附属第二医院);Peking University School and Hospital of Stomatology(北京大学口腔医学院(口腔医院));Medlinker Intelligent and Digital Technology Co., Ltd(医联智数科技有限公司);The University of Queensland(昆士兰大学);Beijing Xicheng District Health Care Hospital for Mothers and Children(北京市西城区妇幼保健院)
Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Peking University(北京大学);University of Chinese Academy of Sciences(中国科学院大学);Shanghai University(上海大学)
Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Models
University of Chinese Academy of Sciences(中国科学院大学);Chinese Information Processing Laboratory(中文信息处理实验室);State Key Laboratory of Computer Science(计算机科学国家重点实验室);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)
ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM
Institute of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models
University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心);Helmholtz Munich(亥姆霍兹慕尼黑中心);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);University of Virginia(弗吉尼亚大学);University of British Columbia(不列颠哥伦比亚大学);Harvard Medical School(哈佛医学院);Technical University of Munich(慕尼黑工业大学)