arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.07020cs.CLcs.AI
收藏

TurkBench:评估土耳其大型语言模型的基准测试

TurkBench: A Benchmark for Evaluating Turkish Large Language Models

Computer Sci. Dpt., Bilkent Uni.(计算机科学系,比尔肯特大学);Mathematics Dpt., Middle East Technical University(数学系,中东部技术大学);Turkcell AI(Turkcell人工智能);Sociology Dpt., Hacettepe University(社会学系,哈恰塔尔佩大学);Computer Engineering Dpt., Bogazici University(计算机工程系,博资基大学);Linguistics Dpt., Istanbul University(语言学系,伊斯坦布尔大学);Turkish Lang. Dpt., Middle East Technical University(土耳其语言系,中东部技术大学)
Çağrı Toraman, Ahmet Kaan Sever, Ayse Aysu Cengiz, Elif Ecem Arslan, Görkem Sevinç, Mete Mert Birdal, Yusuf Faruk Güldemir, Ali Buğra Kanburoğlu, Sezen Felekoğl… 展开作者

Çağrı Toraman, Ahmet Kaan Sever, Ayse Aysu Cengiz, Elif Ecem Arslan, Görkem Sevinç, Mete Mert Birdal, Yusuf Faruk Güldemir, Ali Buğra Kanburoğlu, Sezen Felekoğlu, Osman Gürlek, Sarp Kantar, Birsen Şahin Kütük, Büşra Tufan, Elif Genç, Serkan Coşkun, Gupse Ekin Demir, Muhammed Emin Arayıcı, Olgun Dursun, Onur Gungor, Susan Üsküdarlı, Abdullah Topraksoy, Esra Darıcı

AI总结:

TurkBench是一个用于评估土耳其大型语言模型能力的综合基准测试,包含21个子任务和6大类别,旨在帮助研究人员改进模型性能。

摘要:

随着大型语言模型发展迅速,对全面且语言特定的评估基准的需求变得至关重要。尽管在评估英语模型方面取得了显著进展,但其他语言,尤其是具有独特语言特征的土耳其语,的基准测试仍不完善。我们的研究引入了TurkBench,一个全面的基准测试,旨在评估生成式大型语言模型在土耳其语中的能力。TurkBench包含8151个数据样本,涵盖21个不同的子任务。这些任务被组织在六个主要评估类别下:知识、语言理解、推理、内容审查、土耳其语语法和词汇,以及遵循指示。广泛的任务范围和文化相关数据将为研究人员和开发者提供有价值的工具,用于评估其模型并发现改进领域。我们进一步在https://huggingface.co/turkbench发布我们的基准测试以供在线提交。

英文摘要:

With the recent surge in the development of large language models, the need for comprehensive and language-specific evaluation benchmarks has become critical. While significant progress has been made in evaluating English-language models, benchmarks for other languages, particularly those with unique linguistic characteristics such as Turkish, remain less developed. Our study introduces TurkBench, a comprehensive benchmark designed to assess the capabilities of generative large language models in the Turkish language. TurkBench involves 8,151 data samples across 21 distinct subtasks. These are organized under six main categories of evaluation: Knowledge, Language Understanding, Reasoning, Content Moderation, Turkish Grammar and Vocabulary, and Instruction Following. The diverse range of tasks and the culturally relevant data would provide researchers and developers with a valuable tool for evaluating their models and identifying areas for improvement. We further publish our benchmark for online submissions at https://huggingface.co/turkbench

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑