SKATE，一种可扩展的锦标赛评估：较弱的LLM通过可验证的挑战区分更强的LLM
SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challenges
AI总结：
SKATE通过让LLM相互生成和解决可验证任务，实现可扩展的评估框架，揭示模型间的细微能力差异。
摘要：
评估基础模型的能力和风险至关重要，但当前的方法需要大量的领域专业知识，阻碍了其可扩展性，因为这些模型迅速发展。我们介绍了SKATE：一种新的评估框架，其中大型语言模型（LLM）通过为彼此生成和解决问题来竞争。我们的核心见解是将评估视为一场游戏：模型既是任务设置者又是解决问题者，被激励去创造问题以突出自身优势并暴露他人的弱点。SKATE提供了几个关键优势，平衡了可扩展性、开放性和客观性。它是完全自动化的，数据无关，可扩展，不需要任何人类输入或领域专业知识。通过使用可验证的任务而不是LLM裁判，评分是客观的。与领域有限的程序生成基准（例如国际象棋或空间推理）不同，让LLM创造性地提出挑战使开放性和可扩展性评估成为可能。作为概念验证，我们引入了LLM设置的代码输出预测（COP）挑战作为一种可验证且可扩展的框架，用于测试我们的方法。使用基于TrueSkill的排名系统，我们评估了六个前沿LLM，并发现：（1）较弱的模型可以可靠地区分并评分更强的模型；（2）基于LLM的系统能够表现出自我偏好行为，生成与自身能力一致的问题；（3）SKATE会自动揭示模型之间的细微能力差异。我们的发现是迈向通用、可扩展的评估框架的重要一步，这些框架能够与LLM的进步同步。
英文摘要：
Evaluating the capabilities and risks of foundation models is paramount, yet current methods demand extensive domain expertise, hindering their scalability as these models rapidly evolve. We introduce SKATE: a novel evaluation framework in which large language models (LLMs) compete by generating and solving verifiable tasks for one another. Our core insight is to treat evaluation as a game: models act as both task-setters and solvers, incentivized to create questions which highlight their own strengths while exposing others' weaknesses. SKATE offers several key advantages, balancing scalability, open-endedness, and objectivity. It is fully automated, data-free, and scalable, requiring no human input or domain expertise. By using verifiable tasks rather than LLM judges, scoring is objective. Unlike domain-limited programmatically-generated benchmarks (e.g. chess-playing or spatial reasoning), having LLMs creatively pose challenges enables open-ended and scalable evaluation. As a proof of concept, we introduce LLM-set code-output-prediction (COP) challenges as a verifiable and extensible framework in which to test our approach. Using a TrueSkill-based ranking system, we evaluate six frontier LLMs and find that: (1) weaker models can reliably differentiate and score stronger ones, (2) LLM-based systems are capable of self-preferencing behavior, generating questions that align with their own capabilities, and (3) SKATE automatically surfaces fine-grained capability differences between models. Our findings are an important step towards general, scalable evaluation frameworks which can keep pace with LLM progress.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
