最大化你的扩散：基于扩散的控制中的奖励最大化与对齐研究
Maximize Your Diffusion: A Study into Reward Maximization and Alignment for Diffusion-based Control
AI总结：
本文研究基于扩散的控制中的奖励最大化问题，扩展并统一强化学习奖励对齐、DPO、监督微调和级联扩散四种微调方法，在离线 RL 控制任务中取得实证提升。
摘要：
基于扩散的规划、学习和控制方法构成了强大且富有表达力的决策解决方案中一个颇具前景的分支。鉴于人们日益增长的兴趣，此类方法在过去几年中经历了大量改进。然而，尽管取得了这些进展，现有方法在研究决策过程中用于奖励最大化的通用方法方面仍存在局限。在本工作中，我们研究面向控制应用的微调方法扩展。具体而言，我们探讨了四种微调方法的扩展和多种设计选择：通过强化学习进行奖励对齐、直接偏好优化、监督微调和级联扩散。我们优化了它们的使用方式，以将这些相互独立的努力融合为一个统一范式。我们展示了此类提议在离线 RL 设置中的效用，并在丰富的一系列控制任务上证明了实证改进。
英文摘要：
Diffusion-based planning, learning, and control methods present a promising branch of powerful and expressive decision-making solutions. Given the growing interest, such methods have undergone numerous refinements over the past years. However, despite these advancements, existing methods are limited in their investigations regarding general methods for reward maximization within the decision-making process. In this work, we study extensions of fine-tuning approaches for control applications. Specifically, we explore extensions and various design choices for four fine-tuning approaches: reward alignment through reinforcement learning, direct preference optimization, supervised fine-tuning, and cascading diffusion. We optimize their usage to merge these independent efforts into one unified paradigm. We show the utility of such propositions in offline RL settings and demonstrate empirical improvements over a rich array of control tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
