TurkBench:评估土耳其大型语言模型的基准测试
TurkBench: A Benchmark for Evaluating Turkish Large Language Models
AI总结:
TurkBench是一个用于评估土耳其大型语言模型能力的综合基准测试,包含21个子任务和6大类别,旨在帮助研究人员改进模型性能。
摘要:
随着大型语言模型发展迅速,对全面且语言特定的评估基准的需求变得至关重要。尽管在评估英语模型方面取得了显著进展,但其他语言,尤其是具有独特语言特征的土耳其语,的基准测试仍不完善。我们的研究引入了TurkBench,一个全面的基准测试,旨在评估生成式大型语言模型在土耳其语中的能力。TurkBench包含8151个数据样本,涵盖21个不同的子任务。这些任务被组织在六个主要评估类别下:知识、语言理解、推理、内容审查、土耳其语语法和词汇,以及遵循指示。广泛的任务范围和文化相关数据将为研究人员和开发者提供有价值的工具,用于评估其模型并发现改进领域。我们进一步在https://huggingface.co/turkbench发布我们的基准测试以供在线提交。
英文摘要:
With the recent surge in the development of large language models, the need for comprehensive and language-specific evaluation benchmarks has become critical. While significant progress has been made in evaluating English-language models, benchmarks for other languages, particularly those with unique linguistic characteristics such as Turkish, remain less developed. Our study introduces TurkBench, a comprehensive benchmark designed to assess the capabilities of generative large language models in the Turkish language. TurkBench involves 8,151 data samples across 21 distinct subtasks. These are organized under six main categories of evaluation: Knowledge, Language Understanding, Reasoning, Content Moderation, Turkish Grammar and Vocabulary, and Instruction Following. The diverse range of tasks and the culturally relevant data would provide researchers and developers with a valuable tool for evaluating their models and identifying areas for improvement. We further publish our benchmark for online submissions at https://huggingface.co/turkbench
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。