ToTRL: 通过解谜游戏解锁大语言模型树状思维推理潜力
ToTRL: Unlock LLM Tree-of-Thoughts Reasoning Potential through Puzzles Solving
AI总结：
ToTRL通过解谜游戏训练，提升大语言模型的树状思维推理能力与效率。
摘要：
大型语言模型（LLMs）在推理能力方面表现出显著的潜力，特别是在通过长链式思维（CoT）过程进行推理时，这种能力可以通过强化学习（RL）来激发。然而，长时间的CoT推理存在局限性，主要问题在于输出冗长，由于过度的自我反思。这些LLM的推理过程往往似乎遵循试错方法，而不是系统性的逻辑推导。相比之下，树状思维（ToT）提供了一种概念上更先进的方法，通过将推理建模为树结构内的探索。这种推理结构有助于并行生成和评估多个推理分支，允许主动识别、评估和修剪不产生成果的路径。这一过程可能有助于提高性能并减少令牌成本。在基于LLM的长CoT能力的基础上，我们引入了树状思维强化学习（ToTRL），一种新颖的在线强化学习框架，具有基于规则的奖励机制。ToTRL旨在引导LLM开发基于顺序CoT策略的并行ToT策略。此外，我们在ToTRL训练过程中将LLM作为解谜游戏的玩家。解谜游戏本质上需要探索相互依赖的选择并管理多个约束，这要求构建和探索思维树，为培养ToT推理能力提供了具有挑战性的任务。我们的实证评估表明，我们的ToTQwen3-8B模型在复杂推理任务上通过我们的ToTRL训练实现了显著的性能提升和推理效率提高。
英文摘要：
Large language models (LLMs) demonstrate significant reasoning capabilities, particularly through long chain-of-thought (CoT) processes, which can be elicited by reinforcement learning (RL). However, prolonged CoT reasoning presents limitations, primarily verbose outputs due to excessive introspection. The reasoning process in these LLMs often appears to follow a trial-and-error methodology rather than a systematic, logical deduction. In contrast, tree-of-thoughts (ToT) offers a conceptually more advanced approach by modeling reasoning as an exploration within a tree structure. This reasoning structure facilitates the parallel generation and evaluation of multiple reasoning branches, allowing for the active identification, assessment, and pruning of unproductive paths. This process can potentially lead to improved performance and reduced token costs. Building upon the long CoT capability of LLMs, we introduce tree-of-thoughts RL (ToTRL), a novel on-policy RL framework with a rule-based reward. ToTRL is designed to guide LLMs in developing the parallel ToT strategy based on the sequential CoT strategy. Furthermore, we employ LLMs as players in a puzzle game during the ToTRL training process. Solving puzzle games inherently necessitates exploring interdependent choices and managing multiple constraints, which requires the construction and exploration of a thought tree, providing challenging tasks for cultivating the ToT reasoning capability. Our empirical evaluations demonstrate that our ToTQwen3-8B model, trained with our ToTRL, achieves significant improvement in performance and reasoning efficiency on complex reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
