ChessQA：评估大型语言模型的国际象棋理解能力
ChessQA: Evaluating Large Language Models for Chess Understanding
AI总结：
针对现有LLM国际象棋能力评估临时且范围窄的问题，提出涵盖五类任务的动态综合基准ChessQA，评估发现当代LLM在各类别均有缺陷，并将开放相关资源支撑后续研究。
摘要：
国际象棋为评估大型语言模型（LLM）的推理、建模和抽象能力提供了理想的测试平台，因为它具有明确的结构和客观的真值，同时涵盖广泛的技能水平区间。然而，现有针对LLM国际象棋能力的评估是临时性的，且范围狭窄，难以准确衡量LLM的国际象棋理解能力，以及该能力如何随模型规模、后训练方法或架构选择发生变化。我们提出了ChessQA，这是一个综合性基准，可从五个任务类别（结构类、定式类、短战术类、局面判断类和语义类）评估LLM的国际象棋理解能力，这五个类别大致对应棋手积累国际象棋知识过程中逐步掌握的抽象层级：从理解基本规则、学习战术定式，到正确计算战术、评估局面，再到语义化描述高层概念。通过这种方式，ChessQA能更全面地刻画国际象棋能力与理解水平，远超以往仅做简单走子质量评估的方式，并为模型诊断与比较提供了可控、一致的环境。此外，ChessQA本质上是动态的，其提示词、答案密钥和构建脚本可随模型能力提升而迭代。通过评估一系列当代LLM，我们发现模型在所有五个类别中都存在持续性缺陷，并提供了按类别划分的结果与错误分析。我们将发布代码、定期更新的数据集以及公开排行榜，以支持后续研究。
英文摘要：
Chess provides an ideal testbed for evaluating the reasoning, modeling, and abstraction capabilities of large language models (LLMs), as it has well-defined structure and objective ground truth while admitting a wide spectrum of skill levels. However, existing evaluations of LLM ability in chess are ad hoc and narrow in scope, making it difficult to accurately measure LLM chess understanding and how it varies with scale, post-training methodologies, or architecture choices. We present ChessQA, a comprehensive benchmark that assesses LLM chess understanding across five task categories (Structural, Motifs, Short Tactics, Position Judgment, and Semantic), which approximately correspond to the ascending abstractions that players master as they accumulate chess knowledge, from understanding basic rules and learning tactical motifs to correctly calculating tactics, evaluating positions, and semantically describing high-level concepts. In this way, ChessQA captures a more comprehensive picture of chess ability and understanding, going significantly beyond the simple move quality evaluations done previously, and offers a controlled, consistent setting for diagnosis and comparison. Furthermore, ChessQA is inherently dynamic, with prompts, answer keys, and construction scripts that can evolve as models improve. Evaluating a range of contemporary LLMs, we find persistent weaknesses across all five categories and provide results and error analyses by category. We will release the code, periodically refreshed datasets, and a public leaderboard to support further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
