arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.01683cs.CLcs.AI
收藏

LLM驱动的基准工厂:可靠、通用且高效

LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient

School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Xiaohongshu Inc(小红书科技有限公司)

Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li

AI总结:

本文提出BenchMaker,通过四维十标准评估框架和一系列可靠性增强方法,构建了可靠、通用且高效的LLM基准自动生成器,性能媲美人工标注且成本极低。

摘要:

大型语言模型(LLMs)的快速发展导致了模型供给和应用需求的双重激增。为了促进二者之间的有效匹配,业界广泛需要可靠、通用且高效的基准生成器。然而,人工标注者受限于效率低下,而当前的LLM基准生成器不仅缺乏通用性,还在可靠性方面存在不足,因为它们缺少用于验证和优化的全面评估框架。为填补这一空白,我们首先提出了一个自动化且无偏的评估框架,围绕四个维度和十项标准构建。在该框架下,我们仔细分析了直接提示LLMs作为通用基准生成器的优势与不足。为提升可靠性,我们引入了一系列方法来解决已识别的不足,并将其整合为BenchMaker。跨多个LLMs和任务的实验证实,BenchMaker在所有指标上均达到与人工标注基准相当或更优的性能,凸显了其通用性和可靠性。更重要的是,它在12个LLMs上产生高度一致的评估结果(与MMLU-Pro的皮尔逊相关系数为0.967),而每个样本仅需0.005美元和0.38分钟。

英文摘要:

The rapid advancement of large language models (LLMs) has led to a surge in both model supply and application demands. To facilitate effective matching between them, reliable, generic and efficient benchmark generators are widely needed. However, human annotators are constrained by inefficiency, and current LLM benchmark generators not only lack generalizability but also struggle with limited reliability, as they lack a comprehensive evaluation framework for validation and optimization. To fill this gap, we first propose an automated and unbiased evaluation framework, structured around four dimensions and ten criteria. Under this framework, we carefully analyze the advantages and weaknesses of directly prompting LLMs as generic benchmark generators. To enhance the reliability, we introduce a series of methods to address the identified weaknesses and integrate them as BenchMaker. Experiments across multiple LLMs and tasks confirm that BenchMaker achieves superior or comparable performance to human-annotated benchmarks on all metrics, highlighting its generalizability and reliability. More importantly, it delivers highly consistent evaluation results across 12 LLMs (0.967 Pearson correlation against MMLU-Pro), while taking only $0.005 and 0.38 minutes per sample.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑