用梯度对抗不确定性：通过扩散分数匹配进行离线强化学习
Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching
AI总结：
本文提出分数引导规划（SGP），利用分数匹配学习到数据距离的梯度，实现离线强化学习中的高效一阶规划，解决高维问题中零阶方法无法扩展和集成方法陷入局部最小值的问题。
摘要：
基于梯度的方法在高维空间中实现了高效的搜索能力。然而，为了在离线优化范式（如离线强化学习（RL）或模仿学习（IL））中有效应用它们，我们需要更仔细地考虑不确定性估计如何与试图最小化这些不确定性的二阶方法相互作用。我们将到数据的平滑距离作为一种不确定性度量进行研究，并声称它有两个有益的性质：（i）它允许试图最小化不确定性的基于梯度的方法在平滑退火时驱动迭代到数据；（ii）它有助于用Lipschitz常数分析模型偏差。由于到数据的距离在线计算可能昂贵，我们考虑需要摊销这种计算的环境。然而，我们不学习距离本身，而是提出直接学习其梯度作为一阶优化器的预言机。我们表明，通过利用到数据的距离与数据似然之间的等价性，这些梯度可以用分数匹配技术高效地学习。利用这一见解，我们提出了分数引导规划（SGP），一种用于离线强化学习的规划算法，它利用分数匹配实现在高维问题中的一阶规划，而零阶方法无法扩展，集成方法无法克服局部最小值。网站：https://sites.google.com/view/score-guided-planning/home
英文摘要：
Gradient-based methods enable efficient search capabilities in high dimensions. However, in order to apply them effectively in offline optimization paradigms such as offline Reinforcement Learning (RL) or Imitation Learning (IL), we require a more careful consideration of how uncertainty estimation interplays with first-order methods that attempt to minimize them. We study smoothed distance to data as an uncertainty metric, and claim that it has two beneficial properties: (i) it allows gradient-based methods that attempt to minimize uncertainty to drive iterates to data as smoothing is annealed, and (ii) it facilitates analysis of model bias with Lipschitz constants. As distance to data can be expensive to compute online, we consider settings where we need amortize this computation. Instead of learning the distance however, we propose to learn its gradients directly as an oracle for first-order optimizers. We show these gradients can be efficiently learned with score-matching techniques by leveraging the equivalence between distance to data and data likelihood. Using this insight, we propose Score-Guided Planning (SGP), a planning algorithm for offline RL that utilizes score-matching to enable first-order planning in high-dimensional problems, where zeroth-order methods were unable to scale, and ensembles were unable to overcome local minima. Website: https://sites.google.com/view/score-guided-planning/home
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
