扩散对齐超越KL：方差最小化作为有效的策略优化器
Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimiser
AI总结：
本文提出方差最小化策略优化方法，通过最小化对数重要权重的方差来实现扩散对齐，超越传统KL优化，提供新的设计方向。
摘要：
扩散对齐通过适应预训练的扩散模型来采样奖励倾斜的分布，沿着去噪轨迹进行。这一过程自然地允许序贯蒙特卡洛（SMC）的解释，其中去噪模型作为提案，奖励引导诱导重要权重。受此观点启发，我们引入了方差最小化策略优化（VMPO），将扩散对齐表述为最小化对数重要权重的方差，而不是直接优化基于KL的客观函数。我们证明了方差目标由奖励倾斜的目标分布最小化，并且在on-policy采样下，其梯度与标准KL对齐的梯度一致。这一视角为理解扩散对齐提供了共同的视角。在不同的潜在函数和方差最小化策略选择下，VMPO恢复了各种现有方法，同时也提出了超越KL的新设计方向。
英文摘要：
Diffusion alignment adapts pretrained diffusion models to sample from reward-tilted distributions along the denoising trajectory. This process naturally admits a Sequential Monte Carlo (SMC) interpretation, where the denoising model acts as a proposal and reward guidance induces importance weights. Motivated by this view, we introduce Variance Minimisation Policy Optimisation (VMPO), which formulates diffusion alignment as minimising the variance of log importance weights rather than directly optimising a Kullback-Leibler (KL) based objective. We prove that the variance objective is minimised by the reward-tilted target distribution and that, under on-policy sampling, its gradient coincides with that of standard KL-based alignment. This perspective offers a common lens for understanding diffusion alignment. Under different choices of potential functions and variance minimisation strategies, VMPO recovers various existing methods, while also suggesting new design directions beyond KL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
