arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.30104cs.CL
收藏

SEAL: 饱和基准能否通过LLM作为元裁判得以复兴?

SEAL: Can Saturated Benchmarks Be Revived by LLM-as-a-Meta-Judge?

ByteDance Inc.(字节跳动公司);City University of Hong Kong(香港城市大学)

Jiamin Chen, Yidi Wu, Qiexiang Wang, Qianben Chen, Yuchen Li, Yansen Zhang, Xiaokun Zhang, Wangchunshu Zhou, Chen Ma

AI总结:

提出SEAL协议,通过自适应LLM元裁判从饱和基准中提取潜在排名信号,在代码生成、数学推理等任务上以更少调用实现高排名准确率。

摘要:

广泛使用的语言模型基准日益饱和,前沿系统常获得标准指标无法区分的接近分数。我们不构建更难的替代方案,而是探究是否可以通过改进对相同候选输出的评估来使现有任务重新具有信息量。因此,我们提出了带自适应LLM元裁判的种子淘汰法,这是一种自我改进的评估协议,用于从饱和基准中提取潜在排名信号。SEAL将候选输出种子化为单淘汰赛,并通过任务级原则和自改进检查表标准评估每场比赛。我们在涵盖代码生成、数学推理、知识密集型问答和工具使用智能体任务完成的多个饱和基准上评估SEAL。在这些设置中,SEAL改善了排名准确性与延迟之间的权衡,与完全成对评判相比达到了0.83-1.00的Spearman一致性和4/4的top-1一致性,同时每个任务仅需11.89次调用,而完全成对评估需要28.00次。

英文摘要:

Widely used language-model benchmarks are increasingly saturated, with frontier systems often receiving near-tied scores that standard metrics cannot resolve. Rather than constructing harder alternatives, we ask whether existing tasks can be made informative again through improved evaluation over the same candidate outputs. Therefore, we present Seeded Elimination with Adaptive LLM-as-a-Meta-Judge, a self-improving evaluation protocol for extracting latent ranking signal from saturated benchmarks. SEAL seeds candidate outputs into a single elimination and evaluates each match with task-level principles plus self-improving checklist criteria. We evaluate SEAL on multiple saturated benchmarks covering code generation, mathematical reasoning, knowledge-intensive question answering, and tool-use agent task completion. Across these settings, SEAL improves the ranking-accuracy--latency trade-off over competing protocols, attaining 0.83--1.00 Spearman agreement with full pairwise judging and 4/4 top-1 agreement, while requiring only 11.89 calls per task compared with 28.00 for full pairwise evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑