二阶平滑规划与最优传输贝尔曼平滑
Second-Order Smooth Planning with Optimal-Transport Bellman Smoothing
浏览论文内容
中文总结 AI 辅助
针对生成模型下的规划问题，提出基于最优传输平滑贝尔曼备份的二阶规划器SecondOrderSmoothCruiser，将预言机复杂度从一阶的$\widetilde O(\varepsilon^{-4})$降至$\widetilde O(\varepsilon^{-3})$。
中文摘要 AI 辅助
使用生成模型进行规划旨在以尽可能少的模拟器调用次数来估计状态的价值。SmoothCruiser通过利用熵正则化贝尔曼备份的平滑性，实现了问题无关的复杂度$\widetilde O(\varepsilon^{-4})$，但其估计器仅为一阶。我们证明，SmoothCruiser型规划器的样本复杂度指数由局部泰勒余项阶数$\beta$决定，得到预言机复杂度$\widetilde O(\varepsilon^{-(2+2/(\beta-1))})$：一阶情形$\beta=2$恢复SmoothCruiser，而二阶/三次余项$\beta=3$产生$\widetilde O(\varepsilon^{-3})$。我们通过作用在动作分布上的最优传输平滑贝尔曼备份达到该区域，该备份具有闭式解、策略梯度和Lipschitz Hessian，其二次修正项允许无偏叉积估计器。由此得到的SecondOrderSmoothCruiser在固定OT参数下达到$\widetilde O(\varepsilon^{-3})$预言机复杂度，并通过显式正则化偏差界将OT、熵正则化和无正则化目标联系起来。
英文摘要
Planning with a generative model aims to estimate the value of a state using as few simulator calls as possible. SmoothCruiser achieves problem-independent complexity $\widetilde O(\varepsilon^{-4})$ by exploiting the smoothness of the entropy-regularized Bellman backup, but its estimator is only first-order. We show that the sample-complexity exponent of SmoothCruiser-type planners is governed by the order $β$ of the local Taylor remainder, giving oracle complexity $\widetilde O(\varepsilon^{-(2+2/(β-1))})$: the first-order case $β=2$ recovers SmoothCruiser, while a second-order/cubic remainder $β=3$ yields $\widetilde O(\varepsilon^{-3})$. We reach this regime with an optimal-transport-smoothed Bellman backup over action distributions, which has a closed form, a policy gradient, and a Lipschitz Hessian, and whose quadratic correction admits an unbiased cross-product estimator. The resulting SecondOrderSmoothCruiser achieves $\widetilde O(\varepsilon^{-3})$ oracle complexity for fixed OT parameters, and we relate the OT, entropy-regularized, and unregularized objectives through explicit regularization-bias bounds.
发表机构
- Hanoi University of Science and Technology（河内理工大学）
机构由 AI 辅助整理，请以论文原文为准。
