HLE-Verified: 人类最后考试的系统验证与结构化修订
HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam
AI总结:
HLE-Verified通过系统验证和结构化修订,减少噪声并提高模型评估的准确性。
摘要:
人类最后考试(HLE)已成为评估前沿大语言模型在具有挑战性的多领域问题上的表现的广泛使用的基准。然而,社区主导的分析引发了担忧,即HLE包含非平凡数量的噪声项,这些项可能会偏移评估结果并扭曲跨模型比较。为了解决这一挑战,我们引入了HLE-Verified,这是HLE的经过验证和修订的版本,具有透明的验证协议和细粒度的错误分类。我们的构建遵循一个两阶段的验证和修复工作流程,最终产生一个认证的基准。在第一阶段,每个项目通过领域专家审查和基于模型的交叉检查进行二元验证,得出668个验证过的项目。在第二阶段,存在错误但可以修复的项目在严格约束下进行修订,以保持原始评估意图,通过双独立专家修复、模型辅助审计和最终裁决,产生1,143个修订和认证的项目。剩下的689个项目作为文档化的不确定集发布,带有明确的不确定性来源和专家标签,供未来改进。我们对八个最先进的语言模型在HLE和HLE-Verified上进行评估,发现HLE-Verified上的平均绝对准确率提高了7-10个百分点。改进在原始问题陈述和/或参考答案存在错误的项目上尤为明显,提高了30-40个百分点。我们的分析进一步揭示了模型信心与问题陈述或参考答案中存在错误之间的强关联,支持了我们修订的有效性。总体而言,HLE-Verified通过减少注释噪声,提高了HLE类型的评估,使模型能力的测量更加忠实。数据可在:https://huggingface.co/datasets/skylenage/HLE-Verified获取。
英文摘要:
Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions. However, community-led analyses have raised concerns that HLE contains a non-trivial number of noisy items, which can bias evaluation results and distort cross-model comparisons. To address this challenge, we introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy. Our construction follows a two-stage validation-and-repair workflow resulting in a certified benchmark. In Stage I, each item undergoes binary validation of the problem and final answer through domain-expert review and model-based cross-checks, yielding 668 verified items. In Stage II, flawed but fixable items are revised under strict constraints preserving the original evaluation intent, through dual independent expert repairs, model-assisted auditing, and final adjudication, resulting in 1,143 revised-and-certified items. The remaining 689 items are released as a documented uncertain set with explicit uncertainty sources and expertise tags for future refinement. We evaluate eight state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points on HLE-Verified. The improvement is particularly pronounced on items where the original problem statement and/or reference answer is erroneous, with gains of 30--40 percentage points. Our analyses further reveal a strong association between model confidence and the presence of errors in the problem statement or reference answer, supporting the effectiveness of our revisions. Overall, HLE-Verified improves HLE-style evaluations by reducing annotation noise and enabling more faithful measurement of model capabilities. Data is available at: https://huggingface.co/datasets/skylenage/HLE-Verified
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。