结合树搜索的策略梯度：通过前瞻避免局部最优
Policy Gradient with Tree Search: Avoiding Local Optimas through Lookahead
AI总结：
本文提出融合 $m$ 步树搜索前瞻的 PGTS 方法，从理论上证明增加搜索深度可减少不良驻点并改善最坏情况性能，且在多种 MDP 实验中优于标准 PG。
摘要：
强化学习中的经典策略梯度（PG）方法经常收敛到次优局部最优，这一挑战在大型或复杂环境中更为严重。本文研究 Policy Gradient with Tree Search（PGTS），该方法集成了一种 $m$ 步前瞻机制以增强策略优化。我们给出的理论分析表明，增加树搜索深度 $m$ 会单调缩减不良驻点集合，从而改善任何所得驻留策略的最坏情况性能。关键在于，我们的分析适用于策略更新被限制在当前策略访问过的状态、而非要求在整个状态空间上更新的实际情形。在包括 Ladder、Tightrope 和 Gridworld 环境在内的多种 MDP 结构上的实证评估表明，PGTS 能够展现出“远见”，穿越困难的奖励景观，逃脱标准 PG 会失败的局部陷阱，并获得更优解。
英文摘要：
Classical policy gradient (PG) methods in reinforcement learning frequently converge to suboptimal local optima, a challenge exacerbated in large or complex environments. This work investigates Policy Gradient with Tree Search (PGTS), an approach that integrates an $m$-step lookahead mechanism to enhance policy optimization. We provide theoretical analysis demonstrating that increasing the tree search depth $m$-monotonically reduces the set of undesirable stationary points and, consequently, improves the worst-case performance of any resulting stationary policy. Critically, our analysis accommodates practical scenarios where policy updates are restricted to states visited by the current policy, rather than requiring updates across the entire state space. Empirical evaluations on diverse MDP structures, including Ladder, Tightrope, and Gridworld environments, illustrate PGTS's ability to exhibit "farsightedness," navigate challenging reward landscapes, escape local traps where standard PG fails, and achieve superior solutions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
