从音素到语义：面向泰米尔语的大语言模型评估
From Phonemes to Meaning: Evaluating Large Language Models on Tamil
AI总结：
针对现有多语言基准难以准确评估低资源形态丰富语言性能的问题，本文构建泰米尔语专属评估基准ILAKKANAM，评估多款大语言模型并揭示其性能差距与潜在机制。
摘要：
大语言模型（LLMs）已在高资源语言的各类任务上展现出强大的泛化能力，但在泰米尔语这类低资源、形态丰富的语言上，其语言能力仍存在大量未探索的空间。现有多语言基准通常依赖翻译自英语的数据集，无法捕捉目标语言的语言与文化细微差异。为填补这一空白，我们推出ILAKKANAM——首个专门面向泰米尔语的语言评估基准，该基准由斯里兰卡中小学泰米尔语科目考试试卷中的820道题目人工整理构建，每道题目均由受过训练的语言学家标注，涵盖5个语言类别与1个事实知识类别，覆盖1至13年级内容以保证广泛的语言覆盖范围。我们采用标准化评估框架对闭源和开源大语言模型展开评估，结果显示Gemini 2.5取得了最高的整体性能，开源模型则相对落后，凸显出语言基础层面的差距。分类别、分年级的分析表明，所有模型在低年级题目上表现良好，但随着语言复杂度提升性能出现明显下滑。此外，我们未观测到模型的整体性能与其识别语言类别的能力之间存在强相关性，这说明模型的性能可能来自语料接触而非真正的理解。
英文摘要：
Large Language Models (LLMs) have shown strong generalization across tasks in high-resource languages; however, their linguistic competence in low-resource and morphologically rich languages such as Tamil remains largely unexplored. Existing multilingual benchmarks often rely on translated English datasets, failing to capture the linguistic and cultural nuances of the target language. To address this gap, we introduce ILAKKANAM, the first Tamil-specific linguistic evaluation benchmark manually curated using 820 questions from Sri Lankan school-level Tamil subject examination papers. Each question is annotated by trained linguists under five linguistic categories and a factual knowledge category, spanning Grades 1--13 to ensure broad linguistic coverage. We evaluate both closed-source and open-source LLMs using a standardized evaluation framework. Our results show that Gemini 2.5 achieves the highest overall performance, while open-source models lag behind, highlighting the gap in linguistic grounding. Category- and grade-wise analyses reveal that all models perform well on lower-grade questions but show a clear decline as linguistic complexity increases. Further, no strong correlation is observed between a model's overall performance and its ability to identify linguistic categories, suggesting that performance may be driven by exposure rather than genuine understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
