ZeroSumEval：通过模型间竞争扩展大语言模型评估
ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competition
AI总结：
针对传统LLM评估方法的缺陷，提出基于零和博弈的动态评估协议ZeroSumEval，通过多类型游戏评估多维度AI能力，实验表明其能有效评估模型，且前沿模型在创意类任务中表现不佳。
摘要：
评估大语言模型（LLMs）的能力传统上依赖静态基准数据集、人类评估或基于模型的评估——这些方法常受过度拟合、高成本和偏见困扰。ZeroSumEval是一种新颖的基于竞争的评估协议，它利用零和博弈，通过抵抗饱和的动态基准来评估LLMs。ZeroSumEval包含多种游戏，包括安全挑战（PyJail）、经典游戏（国际象棋、说谎者骰子、扑克）、知识测试（MathQuiz）和说服挑战（Gandalf、辩论）。这些游戏旨在评估AI的多种能力，如战略推理、规划、知识应用和创造力。基于最近强调游戏化评估对LLMs有效性的研究，ZeroSumEval通过提供标准化且可扩展的框架增强了这些方法。为证明这一点，我们在7种游戏和13个模型上进行了超过7000次模拟的广泛实验。结果显示，尽管GPT和Claude系列的前沿模型能玩常见游戏和回答问题，但它们难以玩需要创建新颖且具挑战性问题的游戏；我们还观察到模型无法可靠地相互越狱（不执行），且在需要创造力的任务中普遍失败。我们在https://github.com/facebookresearch/ZeroSumEval发布了代码。
英文摘要：
Evaluating the capabilities of Large Language Models (LLMs) has traditionally relied on static benchmark datasets, human assessments, or model-based evaluations - methods that often suffer from overfitting, high costs, and biases. ZeroSumEval is a novel competition-based evaluation protocol that leverages zero-sum games to assess LLMs with dynamic benchmarks that resist saturation. ZeroSumEval encompasses a diverse suite of games, including security challenges (PyJail), classic games (Chess, Liar's Dice, Poker), knowledge tests (MathQuiz), and persuasion challenges (Gandalf, Debate). These games are designed to evaluate a range of AI capabilities such as strategic reasoning, planning, knowledge application, and creativity. Building upon recent studies that highlight the effectiveness of game-based evaluations for LLMs, ZeroSumEval enhances these approaches by providing a standardized and extensible framework. To demonstrate this, we conduct extensive experiments with >7000 simulations across 7 games and 13 models. Our results show that while frontier models from the GPT and Claude families can play common games and answer questions, they struggle to play games that require creating novel and challenging questions. We also observe that models cannot reliably jailbreak each other and fail generally at tasks requiring creativity. We release our code at https://github.com/facebookresearch/ZeroSumEval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
