arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.20099cs.AI
收藏

EvalBuilder:大型语言模型能否构建高效基准?

EvalBuilder: Can LLMs Build Efficient Benchmarks?

Meituan(美团);Pennsylvania State University(宾夕法尼亚州立大学);Allen Institute for AI (AI2)(人工智能艾伦研究所);University of Washington(华盛顿大学);University at Buffalo(布法罗纽约州立大学)

Huaisheng Zhu, Hangfan Zhang, Teng Xiao, Siyuan Xu, Shijie Zhou, Zhimeng Guo, Zhengyu Chen, Jingang Wang

AI总结:

本研究提出EvalBuilder框架,将自动基准开发建模为元评估问题,从保真度、效率、鲁棒性三维度评估LLM智能体构建高效基准的能力,为相关研究提供系统测试平台。

摘要:

大型语言模型(LLM)智能体正越来越多地被用于长周期的研究与工程任务,但对这些能力的评估仍成本高昂、耗时且高度依赖人类专业知识。这催生了一个自然的问题:LLM智能体自身能否构建高效的基准,同时保留高成本参考评估的结论?我们通过将自动基准开发建模为元评估问题来研究这一能力,其中智能体被给定一项任务和一个高成本参考基准,需构建一个低成本代理基准,以忠地区分候选模型与解决方案。基于此建模,我们引入EvalBuilder,这是一个用于评估此类基准生成能力的框架,涵盖保真度、效率、鲁棒性三个关键维度。具体而言,我们评估生成的基准是否能保留模型排名、泛化到未见过的模型及新收集的响应,是否能提供校准良好且具区分性的分数,同时大幅降低评估成本。我们还提供了一个迭代环境,智能体可在固定时间预算下优化其基准。我们的框架为理解当前LLM智能体何时能开发出有用的评估器、以及基准构建本身如何实现日益自动化,提供了系统的测试平台与基准。

英文摘要:

Large language model (LLM) agents are increasingly used for long-horizon research and engineering, yet evaluating these capabilities remains expensive, time-consuming, and heavily dependent on human expertise. This motivates a natural question: can LLM agents themselves construct efficient benchmarks that preserve the conclusions of costly reference evaluations? We study this capability by formulating automatic benchmark development as a meta-evaluation problem, where an agent is given a task and an expensive reference benchmark and requested to construct a lower-cost proxy benchmark that faithfully distinguishes among candidate models and solutions. Based on this formulation, we introduce EvalBuilder, a framework for evaluating such benchmark-generation capabilities along three key dimensions: fidelity, efficiency, and robustness. Specifically, we assess whether generated benchmarks preserve model rankings, generalize to held-out models and freshly collected responses, and provide well-calibrated and discriminative scores while substantially reducing evaluation cost. We further provide an iterative environment in which agents can refine their benchmarks under a fixed time budget. Our framework provides a systematic testbed and benchmark for understanding when current LLM agents can develop useful evaluators and how benchmark construction itself can be increasingly automated.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑