从根到奖励：基于强化学习的动态树推理
From Roots to Rewards: Dynamic Tree Reasoning with Reinforcement Learning
AI总结：
针对静态树推理的局限，提出动态强化学习框架，增量构建推理树并学习最优动作策略，兼顾概率严谨性与计算效率，提升复杂问答性能。
摘要：
现代语言模型通过思维链（CoT）推理（Wei等人，2023）和检索增强（Lewis等人，2021）来处理复杂问题，但仍面临错误传播和知识整合的挑战。树结构推理方法，特别是概率树思维（ProbTree）（Cao等人，2023）框架，通过将问题分解为层次结构，并通过对参数化知识和检索知识进行置信度加权聚合来选择答案（Yao等人，2023），从而缓解了这些问题。然而，ProbTree的静态实现引入了两个关键限制：（1）推理树在初始构建阶段是固定的，无法根据中间结果进行动态调整；（2）每个节点需要对所有可能的解决方案策略进行全面评估，导致计算效率低下。我们提出了一种动态强化学习（Sutton和Barto，2018）框架，将基于树的推理转变为自适应过程。我们的方法基于实时置信度估计增量构建推理树，同时学习动作选择（分解、检索或聚合）的最优策略。这保持了ProbTree的概率严谨性，同时通过选择性扩展和聚焦资源分配提高了解决方案质量和计算效率。这项工作为树结构推理建立了一种新范式，平衡了概率框架的可靠性与现实世界问答系统所需的灵活性。代码可在以下网址获取：https://github.com/ahmedehabb/From-Roots-to-Rewards-Dynamic-Tree-Reasoning-with-RL
英文摘要：
Modern language models address complex questions through chain-of-thought (CoT) reasoning (Wei et al., 2023) and retrieval augmentation (Lewis et al., 2021), yet struggle with error propagation and knowledge integration. Tree-structured reasoning methods, particularly the Probabilistic Tree-of-Thought (ProbTree)(Cao et al., 2023) framework, mitigate these issues by decomposing questions into hierarchical structures and selecting answers through confidence-weighted aggregation of parametric and retrieved knowledge (Yao et al., 2023). However, ProbTree's static implementation introduces two key limitations: (1) the reasoning tree is fixed during the initial construction phase, preventing dynamic adaptation to intermediate results, and (2) each node requires exhaustive evaluation of all possible solution strategies, creating computational inefficiency. We present a dynamic reinforcement learning (Sutton and Barto, 2018) framework that transforms tree-based reasoning into an adaptive process. Our approach incrementally constructs the reasoning tree based on real-time confidence estimates, while learning optimal policies for action selection (decomposition, retrieval, or aggregation). This maintains ProbTree's probabilistic rigor while improving both solution quality and computational efficiency through selective expansion and focused resource allocation. The work establishes a new paradigm for treestructured reasoning that balances the reliability of probabilistic frameworks with the flexibility required for real-world question answering systems. Code available at: https://github.com/ahmedehabb/From-Roots-to-Rewards-Dynamic-Tree-Reasoning-with-RL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
