自对弈搜索蒸馏用于大语言模型推理

中文摘要 AI 辅助

提升大语言模型（LLM）的推理能力需要高质量数据，这些数据应暴露困难决策、竞争性替代方案及其后果。数据稀缺源于合成数据质量低和人工标注成本高。我们提出了自对弈搜索蒸馏（SPSD），一种通过在棋盘游戏上训练的类MuZero网络的自对弈来生成超人类合成数据的框架。SPSD利用可执行环境将搜索转化为结构化推理问题。在每个状态，专家识别出首选决策、合理的替代方案、合理的对手回应以及价值估计。通过将自对弈搜索记录转化为超人类思维链，我们使用环境接地监督来训练LLM。尽管仅使用自对弈搜索记录训练，SPSD仍能迁移到未见过的数学问题。在Qwen3-4B-Base上，它将六个数学基准的平均分从24.1提升至36.6，同时将保留游戏的胜率从15%提升至45%。SPSD提供了一种标注高效的方法，用于创建高质量合成数据，以提升LLM在推理任务中的性能。

英文摘要

Improving reasoning abilities in Large Language Models (LLMs) requires high-quality data that exposes difficult decisions, competing alternatives, and their consequences. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling. We introduce Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games. SPSD uses executable environments to turn search into structured reasoning problems. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates. By converting the self-play search records into superhuman chains-of-thought, we train LLMs with environment-grounded supervision. Although trained only on self-play search records, SPSD transfers to unseen mathematics. On Qwen3-4B-Base, it raises the mean over six mathematics benchmarks from 24.1 to 36.6 while increasing the held-out-game win rate from 15% to 45%. SPSD offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.

发表机构

• University of Bologna（博洛尼亚大学）
• University of Edinburgh（爱丁堡大学）
• Huawei Technologies R&D (UK) Ltd.（华为技术研发（英国）有限公司）

机构由 AI 辅助整理，请以论文原文为准。