探索大语言模型在国际象棋测试平台中的推理能力
Explore the Reasoning Capability of LLMs in the Chess Testbed
- UCLA（加州大学洛杉矶分校）
- Microsoft Research（微软研究院）
- MBZUAI（穆罕默德·本·扎耶德人工智能大学）
- University of Toronto（多伦多大学）
- Peking University（北京大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对大语言模型在长期复杂推理（如国际象棋）中的不足，本文提出通过整合专家标注的策略与战术数据（MATE数据集）微调LLaMA-3-8B，实验证明其优于GPT、Claude和Gemini，并发现语言解释能增强推理能力。
AI中文摘要：
推理是人类智能的核心能力。近年来，随着大规模数据集的出现，预训练的大语言模型涌现出新的能力，包括推理。然而，这些模型在长期、复杂的推理任务（如下国际象棋）中仍然存在困难。基于专家棋手采用将长期战略与短期战术相结合的双重方法并辅以语言解释的观察，我们提出通过整合标注的策略和战术来提升大语言模型在国际象棋中的推理能力。具体而言，我们收集了一个名为MATE的数据集，该数据集包含100万个国际象棋局面，每个局面附有由国际象棋专家标注的候选走法，用于策略和战术。我们微调了LLaMA-3-8B模型，并在选择更优国际象棋走法的任务中将其与最先进的商业语言模型进行比较。我们的实验表明，我们的模型表现优于GPT、Claude和Gemini模型。我们发现语言解释能够增强大语言模型的推理能力。
英文摘要：
Reasoning is a central capability of human intelligence. In recent years, with the advent of large-scale datasets, pretrained large language models have emerged with new capabilities, including reasoning. However, these models still struggle with long-term, complex reasoning tasks, such as playing chess. Based on the observation that expert chess players employ a dual approach combining long-term strategic play with short-term tactical play along with language explanation, we propose improving the reasoning capability of large language models in chess by integrating annotated strategy and tactic. Specifically, we collect a dataset named MATE, which consists of 1 million chess positions with candidate moves annotated by chess experts for strategy and tactics. We finetune the LLaMA-3-8B model and compare it against state-of-the-art commercial language models in the task of selecting better chess moves. Our experiments show that our models perform better than GPT, Claude, and Gemini models. We find that language explanations can enhance the reasoning capability of large language models.
