令牌游戏：通过谜题对决评估语言模型推理能力
The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
AI总结：
本文设计了令牌游戏（TTG）评估框架，通过模型自创谜题进行对决，利用Elo评分比较模型能力，验证了10个前沿模型的推理能力，且无需人工参与谜题创作。
摘要：
本文设计了令牌游戏（TTG）评估框架，通过模型自创谜题进行对决，利用Elo评分比较模型能力，验证了10个前沿模型的推理能力，且无需人工参与谜题创作。
英文摘要：
Evaluating the reasoning capabilities of Large Language Models is increasingly challenging as models improve. Human curation of hard questions is highly expensive, especially in recent benchmarks using PhD-level domain knowledge to challenge the most capable models. Even then, there is always a concern about whether these questions test genuine reasoning or if similar problems have been seen during training. Here, we take inspiration from 16th-century mathematical duels to design The Token Games (TTG): an evaluation framework where models challenge each other by creating their own puzzles. We leverage the format of Programming Puzzles - given a function that returns a boolean, find inputs that make it return True - to flexibly represent problems and enable verifying solutions. Using results from pairwise duels, we then compute Elo ratings, allowing us to compare models relative to each other. We evaluate 10 frontier models on TTG, and closely match the ranking from existing benchmarks such as Humanity's Last Exam, spending less than $200 USD and without involving any human effort in creating puzzles. We also find that creating good puzzles is still a highly challenging task for current models. Overall, our work suggests new paradigms for evaluating reasoning that avoid saturation by design, and that allow testing models for other skills like creativity and task creation alongside problem solving.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
