arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02182cs.CL
收藏

在世界各语言中评估大型语言模型的元语言知识

Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages

University of Ljubljana, Faculty of Computer and Information Science(卢布尔雅那大学计算机与信息科学学院);University of Ljubljana, Faculty of Arts(卢布尔雅那大学艺术学院);Jožef Stefan Institute(乔塞夫·斯塔芬研究所)

Tjaša Arčon, Matej Klemen, Marko Robnik-Šikonja, Kaja Dobrovoljc

AI总结:

本研究通过跨语言评估揭示LLM在元语言知识上的局限性,发现其表现受数据可用性和语言资源影响,而非广泛语法能力。

摘要:

大型语言模型(LLMs)通常在语言使用方面进行评估,但它们对语言结构的显性知识仍缺乏深入理解。现有的语言学基准测试集中于狭窄的语言现象,强调资源丰富的语言,并很少测试元语言知识——即对语言结构的显性推理。我们提出了一个跨语言的LLM元语言知识评估,基于世界语言结构图谱(WALS),记录了2660种语言中的192种语言特征。我们将WALS特征转换为自然语言的多项选择题,并在记录的语言中评估模型。使用准确率和宏F1值,并与偶然性和多数类基线进行比较,我们评估了性能并分析了语言领域和语言相关因素的变化。结果显示了有限的元语言知识:GPT-4o表现最佳,但准确率中等(0.367),而开源模型则落后。尽管所有模型都优于偶然性,但它们未能超越多数类基线,表明它们捕捉到了广泛的跨语言模式,但缺乏细粒度的区分。性能因领域而异,部分反映了在线可见性差异。在语言层面,准确性与数字语言地位相关:数字存在和资源更多的语言被更准确地评估,而低资源语言表现较差。对预测因素的分析确认了资源相关指标(维基百科大小、语料库可用性)比地理、基因学或社会语言学因素更具信息量。总体而言,LLM的元语言知识似乎碎片化,主要由数据可用性而非广泛可推广的语法能力塑造。我们发布此基准作为开源数据集,以支持跨语言的评估,并鼓励未来LLM在更大程度上体现全球语言多样性。

英文摘要:

LLMs are routinely evaluated on language use, yet their explicit knowledge about linguistic structure remains poorly understood. Existing linguistic benchmarks focus on narrow phenomena, emphasize high-resource languages, and rarely test metalinguistic knowledge - explicit reasoning about language structure. We present a multilingual evaluation of metalinguistic knowledge in LLMs, based on the World Atlas of Language Structures (WALS), documenting 192 linguistic features across 2,660 languages. We convert WALS features into natural-language multiple-choice questions and evaluate models across documented languages. Using accuracy and macro F1, and comparing to chance and majority-class baselines, we assess performance and analyse variation across linguistic domains and language-related factors. Results show limited metalinguistic knowledge: GPT-4o performs best but achieves moderate accuracy (0.367), while open-source models lag. Although all models perform above chance, they fail to outperform the majority-class baseline, suggesting they capture broad cross-linguistic patterns but lack fine-grained distinctions. Performance varies by domain, partly reflecting differences in online visibility. At the language level, accuracy correlates with digital language status: languages with greater digital presence and resources are evaluated more accurately, while low-resource languages perform worse. Analysis of predictive factors confirms that resource-related indicators (Wikipedia size, corpus availability) are more informative than geographic, genealogical, or sociolinguistic factors. Overall, LLM metalinguistic knowledge appears fragmented and shaped mainly by data availability, rather than broadly generalizable grammatical competence. We release the benchmark as an open-source dataset to support evaluation across languages and encourage greater global linguistic diversity in future LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑