arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.09346cs.CL
收藏

西班牙的数字语言偏见：来自LLMs词形变异的证据

Evaluating Geographic Lexical Knowledge in Large Language Models Across 21 Spanish-Speaking Countries

University of Tokyo（东京大学）

Yoshifumi Kawasaki

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究通过评估LLMs对西班牙语地域词形变异的识别能力，揭示了模型在不同方言表现上的系统性差异，并指出数据量之外的因素影响方言表征。

摘要：

本研究探讨了大型语言模型（LLMs）在捕捉西班牙语地域词形变异方面的程度，西班牙语表现出显著的地域变异。将LLMs视为虚拟受访者，我们通过两种调查式问题格式来探测其方言知识：是或否问题和多项选择问题。为此，我们利用了一个大规模、专家整理的西班牙语词形变异数据库。我们的评估涵盖了21个西班牙语国家中的超过900个词汇项，并在国家和方言区域两个层面进行评估。在两种评估格式中，结果揭示了LLMs在表示西班牙语方言方面存在系统性差异。与西班牙、赤道几内亚、墨西哥及中美洲以及拉普拉塔河相关的词形变异被模型更准确地识别，而智利方言则特别难以被模型区分。重要的是，国家层面的数字资源数量差异并未解释这些表现模式，这表明影响LLMs方言表征的因素超出了数据量。通过提供对地理词形变异的精细、大规模评估，本研究推进了对LLMs方言知识的实证理解，并为西班牙的数字语言偏见讨论提供了新的证据。

英文摘要：

Large language models (LLMs) increasingly serve users across regional varieties, yet their knowledge of geographic lexical variation remains insufficiently evaluated. We investigate country-conditioned lexical usage judgments in Spanish using an expert-curated database covering 934 lexical items across 21 Spanish-speaking countries. Treating ten proprietary and open-weight LLMs as virtual informants, we evaluate two complementary question formats: Yes-No question format (YNQF) and multiple-choice question format (MCQF). The evaluation comprises 10,164 YNQF questions and 16,637 MCQF questions, assessed using F1 and an adjusted Jaccard coefficient, respectively. Results reveal substantial differences across models, countries, and formats. Spain receives the highest country-level score for eight models in YNQF and nine in MCQF; Chile receives the lowest MCQF score for every model. Regression analyses suggest that digital resource volume is generally more strongly associated with YNQF performance, whereas lexical distance from Spain is more strongly associated with MCQF performance. These findings provide a fine-grained diagnostic of uneven regional lexical knowledge and highlight the importance of complementary elicitation formats and evaluation metrics. The benchmark measures explicit usage judgments, motivating separate assessment of region-appropriate generation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑