arXiv 2506.064702025-06-06cs.AIcs.LG

SIGMA：通过兄弟节点引导的蒙特卡洛增强精炼大语言模型推理

SIGMA: Refining Large Language Model Reasoning via Sibling-Guided Monte Carlo Augmentation

Beihang University（北京航空航天大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Leicester（莱斯特大学）；Nanyang Technological University（南洋理工大学）

Yanwei Ren, Haotian Zhang, Fuxiang Wu, Jiayan Qiu, Jiaxing Huang, Baosheng Yu, Liu Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出 SIGMA 框架，利用 MCTS 中被丢弃的兄弟节点进行比较批判与文本反向传播，以少量数据显著提升 LLM 数学推理。

摘要：

仅通过扩大数据集规模来增强大语言模型已开始出现收益递减，使关注点转向数据质量。Monte Carlo Tree Search（MCTS，蒙特卡洛树搜索）已成为生成高质量思维链数据的强大技术，但传统方法通常只保留搜索树中得分最高的轨迹，丢弃往往包含有价值的局部洞见、反复出现的错误模式和替代推理策略的兄弟节点。这种对非最优推理分支的无条件拒绝可能浪费整棵搜索树中的大量信息性数据。我们提出 SIGMA（Sibling Guided Monte Carlo Augmentation，兄弟节点引导的蒙特卡洛增强），这是一种重新整合这些被丢弃的兄弟节点以精炼 LLM 推理的新颖框架。SIGMA 沿每条搜索路径在兄弟节点之间建立语义链接，并应用两阶段精炼：批判模型识别兄弟节点集合中被忽视的优点和缺点，修订模型执行基于文本的反向传播，根据这种比较反馈精炼得分最高的轨迹。通过恢复并放大非最优推理分支中未充分利用但有价值的信号，SIGMA 显著改进了推理轨迹。在具有挑战性的 MATH 基准测试中，我们经 SIGMA 调优的 7B 模型仅使用 30K 个样本即达到 54.92% 的准确率，优于在 590K 个样本上训练的最先进模型。该结果表明，我们的兄弟节点引导优化不仅显著减少了数据使用量，还显著提升了 LLM 推理能力。

英文摘要：

Enhancing large language models by simply scaling up datasets has begun to yield diminishing returns, shifting the spotlight to data quality. Monte Carlo Tree Search (MCTS) has emerged as a powerful technique for generating high-quality chain-of-thought data, yet conventional approaches typically retain only the top-scoring trajectory from the search tree, discarding sibling nodes that often contain valuable partial insights, recurrent error patterns, and alternative reasoning strategies. This unconditional rejection of non-optimal reasoning branches may waste vast amounts of informative data in the whole search tree. We propose SIGMA (Sibling Guided Monte Carlo Augmentation), a novel framework that reintegrates these discarded sibling nodes to refine LLM reasoning. SIGMA forges semantic links among sibling nodes along each search path and applies a two-stage refinement: a critique model identifies overlooked strengths and weaknesses across the sibling set, and a revision model conducts text-based backpropagation to refine the top-scoring trajectory in light of this comparative feedback. By recovering and amplifying the underutilized but valuable signals from non-optimal reasoning branches, SIGMA substantially improves reasoning trajectories. On the challenging MATH benchmark, our SIGMA-tuned 7B model achieves 54.92% accuracy using only 30K samples, outperforming state-of-the-art models trained on 590K samples. This result highlights that our sibling-guided optimization not only significantly reduces data usage but also significantly boosts LLM reasoning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。