指标计算基准:面向大语言模型的可代码验证的复杂指令遵循基准
Metric Calculating Benchmark: Code-Verifiable Complicate Instruction Following Benchmark for Large Language Models
AI总结:
本文提出MCBench基准,通过可代码验证的逐步指令执行字符串匹配指标,客观评估大语言模型的复杂指令遵循能力,并提供三个评估指标和三个变体。
摘要:
近期前沿级大语言模型已经使许多先前困难的基准测试达到饱和,几乎没有留下进一步区分的空间。这一进展凸显了对能够提供客观验证的具有挑战性基准的需求。在本文中,我们介绍了MCBench,一个旨在评估大语言模型能否通过严格遵循逐步指令来执行字符串匹配自然语言处理指标的基准。与先前依赖主观判断或一般推理的基准不同,MCBench提供了客观、确定且可代码验证的评估。这一设置使我们能够系统地测试大语言模型能否保持准确的逐步执行,包括指令遵循、数值计算以及处理中间结果时的长程一致性。为了确保对这些能力的客观评估,我们提供了并行参考代码,用于评估大语言模型输出的准确性。我们提供了三个评估指标和三个基准变体,旨在衡量大语言模型的详细指令理解能力。我们的分析表明,MCBench是评估前沿大语言模型能力的有效且客观的工具。
英文摘要:
Recent frontier-level LLMs have saturated many previously difficult benchmarks, leaving little room for further differentiation. This progress highlights the need for challenging benchmarks that provide objective verification. In this paper, we introduce MCBench, a benchmark designed to evaluate whether LLMs can execute string-matching NLP metrics by strictly following step-by-step instructions. Unlike prior benchmarks that depend on subjective judgments or general reasoning, MCBench offers an objective, deterministic and codeverifiable evaluation. This setup allows us to systematically test whether LLMs can maintain accurate step-by-step execution, including instruction adherence, numerical computation, and long-range consistency in handling intermediate results. To ensure objective evaluation of these abilities, we provide a parallel reference code that can evaluate the accuracy of LLM output. We provide three evaluative metrics and three benchmark variants designed to measure the detailed instruction understanding capability of LLMs. Our analyses show that MCBench serves as an effective and objective tool for evaluating the capabilities of cutting-edge LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。