本文进行了一项为期十一个月的纵向研究,以解决以往关于 Open Ko-LLM Leaderboard 研究的局限性,这些研究依赖于仅五个月的受限观察期的实证研究。通过延长分析持续时间,我们旨在更全面地理解 Korean large language models (LLMs) 开发的进展。我们的研究由三个主要研究问题指导:(1) 随着时间的推移,在 Open Ko-LLM Leaderboard 上提升 LLM 在多样化任务中的性能面临哪些具体挑战?(2) 模型规模如何影响各种基准测试间的任务性能相关性?(3) Open Ko-LLM Leaderboard 上的排行榜排名模式随时间发生了怎样的变化?通过分析此期间的 1,769 个模型,我们的研究全面考察了 LLM 的持续进步和评估框架的不断演变。
英文摘要:
This paper conducts a longitudinal study over eleven months to address the limitations of prior research on the Open Ko-LLM Leaderboard, which have relied on empirical studies with restricted observation periods of only five months. By extending the analysis duration, we aim to provide a more comprehensive understanding of the progression in developing Korean large language models (LLMs). Our study is guided by three primary research questions: (1) What are the specific challenges in improving LLM performance across diverse tasks on the Open Ko-LLM Leaderboard over time? (2) How does model size impact task performance correlations across various benchmarks? (3) How have the patterns in leaderboard rankings shifted over time on the Open Ko-LLM Leaderboard?. By analyzing 1,769 models over this period, our research offers a comprehensive examination of the ongoing advancements in LLMs and the evolving nature of evaluation frameworks.
The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models
School of Informatics, University of Edinburgh(爱丁堡大学信息学院);School of Electronics Engineering and Computer Science, Peking University(北京大学电子工程与计算机科学学院);Hugging Face;Department of Computer Science, University College London(伦敦大学学院计算机系);Together AI