自动化数据集更新以实现大语言模型的可靠与及时评估
Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models
AI总结：
本文提出自动化更新数据集以应对大语言模型基准泄露和评估滞后问题，设计模仿与扩展两种策略，并通过MMLU和BIG-Bench实验验证其稳定性、难度控制能力和细粒度评估效果。
摘要：
大语言模型（LLMs）在各类自然语言基准上取得了令人瞩目的表现，这促使人们不断需要为更大的大语言模型构建更具挑战性的数据集，而这一过程既昂贵又耗时。在本文中，我们提出自动化数据集更新，并针对其在处理基准泄露问题、难度控制和稳定性方面的有效性提供系统性分析。因此，一旦当前基准已被掌握或泄露，我们就可以对其进行更新，以实现及时且可靠的评估。我们提出两种更新策略：1）模仿策略，基于原始数据生成相似样本，保留风格和上下文本质；2）扩展策略，通过借鉴Bloom教育目标分类法，在不同认知层级上进一步扩展现有样本。在更新后的MMLU和BIG-Bench上的大量实验证明了所提策略的稳定性，并发现模仿策略能够有效缓解因基准泄露而导致的高估问题。在高效的模仿策略失效的情况下，我们的扩展策略仍然展现出有前景的结果。此外，通过控制难度，我们可以更好地区分模型的表现，并实现细粒度分析——过难或过易的考试都无法公平判断学生的学习状况。据我们所知，我们是首个自动化更新基准以实现可靠且及时评估的工作。我们的演示排行榜可在https://yingjiahao14.github.io/Automating-DatasetUpdates/查看。
英文摘要：
Large language models (LLMs) have achieved impressive performance across various natural language benchmarks, prompting a continual need to curate more difficult datasets for larger LLMs, which is costly and time-consuming. In this paper, we propose to automate dataset updating and provide systematic analysis regarding its effectiveness in dealing with benchmark leakage issue, difficulty control, and stability. Thus, once the current benchmark has been mastered or leaked, we can update it for timely and reliable evaluation. There are two updating strategies: 1) mimicking strategy to generate similar samples based on original data, preserving stylistic and contextual essence, and 2) extending strategy that further expands existing samples at varying cognitive levels by adapting Bloom's taxonomy of educational objectives. Extensive experiments on updated MMLU and BIG-Bench demonstrate the stability of the proposed strategies and find that the mimicking strategy can effectively alleviate issues of overestimation from benchmark leakage. In cases where the efficient mimicking strategy fails, our extending strategy still shows promising results. Additionally, by controlling the difficulty, we can better discern the models' performance and enable fine-grained analysis neither too difficult nor too easy an exam can fairly judge students' learning status. To the best of our knowledge, we are the first to automate updating benchmarks for reliable and timely evaluation. Our demo leaderboard can be found at https://yingjiahao14.github.io/Automating-DatasetUpdates/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
