arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.03257cs.CLcs.AI
收藏

通过纵向研究理解 LLM 开发:来自 Open Ko-LLM Leaderboard 的见解

Understanding LLM Development Through Longitudinal Study: Insights from the Open Ko-LLM Leaderboard

Korea University(高丽大学);Upstage AI(上stage人工智能公司)

Chanjun Park, Hyeonwoo Kim

AI总结:

本文通过为期11个月的纵向研究,分析了1,769个模型在Open Ko-LLM Leaderboard上的表现,揭示了Korean LLM开发中的性能挑战、模型规模影响及排名演变,弥补了以往短期观察的局限。

摘要:

本文进行了一项为期十一个月的纵向研究,以解决以往关于 Open Ko-LLM Leaderboard 研究的局限性,这些研究依赖于仅五个月的受限观察期的实证研究。通过延长分析持续时间,我们旨在更全面地理解 Korean large language models (LLMs) 开发的进展。我们的研究由三个主要研究问题指导:(1) 随着时间的推移,在 Open Ko-LLM Leaderboard 上提升 LLM 在多样化任务中的性能面临哪些具体挑战?(2) 模型规模如何影响各种基准测试间的任务性能相关性?(3) Open Ko-LLM Leaderboard 上的排行榜排名模式随时间发生了怎样的变化?通过分析此期间的 1,769 个模型,我们的研究全面考察了 LLM 的持续进步和评估框架的不断演变。

英文摘要:

This paper conducts a longitudinal study over eleven months to address the limitations of prior research on the Open Ko-LLM Leaderboard, which have relied on empirical studies with restricted observation periods of only five months. By extending the analysis duration, we aim to provide a more comprehensive understanding of the progression in developing Korean large language models (LLMs). Our study is guided by three primary research questions: (1) What are the specific challenges in improving LLM performance across diverse tasks on the Open Ko-LLM Leaderboard over time? (2) How does model size impact task performance correlations across various benchmarks? (3) How have the patterns in leaderboard rankings shifted over time on the Open Ko-LLM Leaderboard?. By analyzing 1,769 models over this period, our research offers a comprehensive examination of the ongoing advancements in LLMs and the evolving nature of evaluation frameworks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑