arXiv 2504.097142025-04-26cs.CLcs.AI

评估低资源语言基准数据集的质量：以土耳其语为例

Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkish

Middle East Technical University, Computer Engineering Department（中东部技术大学计算机工程系）；Bilkent University, Computer Engineering Department（比尔肯特大学计算机工程系）；Turkcell AI；Hacettepe University, Sociology Department（哈卡特佩大学社会学系）；Istanbul University, Linguistics Department（伊斯坦布尔大学语言学系）；Middle East Technical University, Turkish Language Department（中东部技术大学土耳其语言系）

Ayşe Aysu Cengiz, Ahmet Kaan Sever, Elif Ecem Ümütlü, Naime Şeyma Erdem, Burak Aytan, Büşra Tufan, Abdullah Topraksoy, Esra Darıcı, Cagri Toraman

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究评估了17个土耳其语基准数据集的质量，发现70%未达标准，强调低资源语言数据集亟需更严格的质量控制。

摘要：

依赖从英语或多语言资源翻译或改编的数据集，在语言和文化适宜性方面带来了挑战。本研究通过评估17个常用土耳其语基准数据集的质量，满足了构建稳健且符合文化的基准的需求。使用评估六项标准的综合框架，人工和LLM-judge标注者均提供了详细评估，以识别数据集的优势与不足。结果显示，70%的基准数据集未达到我们的启发式质量标准。技术术语使用的正确性是最强标准，但在所检查的数据集中有85%的标准未得到满足。尽管LLM评判器展现出潜力，但其效果不如人工标注者，特别是在理解文化常识知识和解释流畅且无歧义的文本方面。GPT-4o在语法和技术任务上具有更强的标注能力，而Llama3.3-70B在正确性和文化知识评估方面表现优异。我们的发现强调，在创建和改编低资源语言数据集时迫切需要更严格的质量控制。

英文摘要：

The reliance on translated or adapted datasets from English or multilingual resources introduces challenges regarding linguistic and cultural suitability. This study addresses the need for robust and culturally appropriate benchmarks by evaluating the quality of 17 commonly used Turkish benchmark datasets. Using a comprehensive framework that assesses six criteria, both human and LLM-judge annotators provide detailed evaluations to identify dataset strengths and shortcomings. Our results reveal that 70% of the benchmark datasets fail to meet our heuristic quality standards. The correctness of the usage of technical terms is the strongest criterion, but 85% of the criteria are not satisfied in the examined datasets. Although LLM judges demonstrate potential, they are less effective than human annotators, particularly in understanding cultural common sense knowledge and interpreting fluent, unambiguous text. GPT-4o has stronger labeling capabilities for grammatical and technical tasks, while Llama3.3-70B excels at correctness and cultural knowledge evaluation. Our findings emphasize the urgent need for more rigorous quality control in creating and adapting datasets for low-resource languages.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。