arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.20574cs.CLcs.AI
收藏

Open Ko-LLM Leaderboard:基于Ko-H5 Benchmark的韩语大语言模型评估

Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark

Upstage AI(上 stage 人工智能公司)

Chanjun Park, Hyeonwoo Kim, Dahyun Kim, Seonghwan Cho, Sanghoon Kim, Sukyung Lee, Yungi Kim, Hwalsuk Lee

AI总结:

本研究推出Open Ko-LLM Leaderboard与Ko-H5 Benchmark用于韩语大模型评估,通过私有测试集、泄露分析等验证框架有效性,为拓展多语言LLM评估提供先例。

摘要:

本文介绍了Open Ko-LLM Leaderboard(开放韩语大模型排行榜)和Ko-H5 Benchmark(韩语H5基准),它们是评估韩语大语言模型(LLM)的重要工具。该框架对标英文Open LLM Leaderboard,采用私有测试集,构建了一套稳健的评估体系,已在韩语LLM社区得到良好应用。我们开展了数据泄露分析,验证了私有测试集的优势,同时进行了Ko-H5基准内部的相关性研究及Ko-H5分数的时序分析。此外,我们还通过实证证明,有必要突破固定基准的局限。我们希望Open Ko-LLM Leaderboard能为拓展LLM评估范围、促进语言多样性树立先例。

英文摘要:

This paper introduces the Open Ko-LLM Leaderboard and the Ko-H5 Benchmark as vital tools for evaluating Large Language Models (LLMs) in Korean. Incorporating private test sets while mirroring the English Open LLM Leaderboard, we establish a robust evaluation framework that has been well integrated in the Korean LLM community. We perform data leakage analysis that shows the benefit of private test sets along with a correlation study within the Ko-H5 benchmark and temporal analyses of the Ko-H5 score. Moreover, we present empirical support for the need to expand beyond set benchmarks. We hope the Open Ko-LLM Leaderboard sets precedent for expanding LLM evaluation to foster more linguistic diversity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑