LLM CHESS: 通过国际象棋评估大语言模型的推理与指令遵循能力
LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess
AI总结：
LLM CHESS通过国际象棋评估大语言模型的推理与指令遵循能力，揭示了推理模型与非推理模型的显著差异，并提供了评估框架和公开排行榜。
摘要：
我们引入了LLM CHESS评估框架，旨在通过在国际象棋领域进行扩展的代理互动来探测大型语言模型（LLMs）的推理和指令遵循能力的泛化能力。我们通过与随机对手对弈，利用包括胜率和败率、走法质量、走法合法性、幻觉动作和游戏时长等行为指标，对超过50个开源和闭源模型进行排名。对于部分顶级推理模型，我们通过与技能可变配置的国际象棋引擎对弈，推导出Elo估计值，从而以易于理解的方式进行比较。尽管指令遵循任务的简单性以及对手的弱性，许多最先进的模型仍难以完成游戏或实现一致的胜利。与其他复杂推理任务的基准测试类似，我们的实验揭示了推理模型与非推理模型之间的明显分离。然而，与现有静态基准测试不同，LLM CHESS的随机性和动态性独特地减少了过拟合和记忆化，防止了基准饱和，即使对顶级推理模型也证明困难。为了支持未来评估LLMs推理和指令遵循能力的工作，我们发布了我们的实验框架、公开排行榜和相关游戏的数据集。
英文摘要：
We introduce LLM CHESS, an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs) through extended agentic interaction in the domain of chess. We rank over 50 open and closed source models by playing against a random opponent using a range of behavioral metrics, including win and loss rates, move quality, move legality, hallucinated actions, and game duration. For a subset of top reasoning models, we derive an Elo estimate by playing against a chess engine with variably configured skill, which allows for comparisons between models in an easily understandable way. Despite the simplicity of the instruction-following task and the weakness of the opponent, many state-of-the-art models struggle to complete games or achieve consistent wins. Similar to other benchmarks on complex reasoning tasks, our experiments reveal a clear separation between reasoning and non-reasoning models. However, unlike existing static benchmarks, the stochastic and dynamic nature of LLM CHESS uniquely reduces overfitting and memorization while preventing benchmark saturation, proving difficult even for top reasoning models. To support future work on evaluating reasoning and instruction-following in LLMs, we release our experimental framework, a public leaderboard, and a dataset of associated games.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
