arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.09433cs.CV
收藏

离线偏好优化用于具有噪声跟踪配对的校正流

Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

Zhejiang University(浙江大学);Shanghai Institute for Advanced Study-Zhejiang University(上海先进研究院-浙江大学);Shanghai Institute for Mathematics and Interdisciplinary Sciences(上海数学与交叉科学研究院)

Yunhong Lu, Qichao Wang, Hengyuan Cao, Xiaoyin Xu, Min Zhang

AI总结:

本文提出PNAPO框架,通过保留生成胜败图像的先验噪声,改进校正流的偏好优化,提升稳定性和样本效率。

摘要:

现有文本到图像模型的偏好数据通常只存储最终胜败图像。这种表示对于校正流(RF)模型不足,因其生成过程自然由特定先验噪声样本索引,并遵循近似直线去噪轨迹。相比之下,先前的DPO风格对扩散模型的对齐通常通过独立的正向噪声过程估计轨迹,这可能导致与真实反向动力学不匹配并引入不必要的方差。我们提出Prior Noise-Aware Preference Optimization (PNAPO),一种专门针对校正流的离线对齐框架。PNAPO通过保留生成每个胜败图像所用的配对先验噪声,将标准(提示,胜者,败者)三元组扩展为六元组。利用校正流的直线性质,我们通过噪声-图像插值估计中间状态,这约束了轨迹估计空间并产生更紧的替代目标函数用于偏好优化。此外,我们引入了一种动态正则化策略,根据(i)胜败之间的奖励差距和(ii)训练进度调整DPO正则化,从而提高稳定性和样本效率。在最先进的RF T2I后端上进行的实验表明,PNAPO在提升偏好度量的同时显著减少了训练计算。

英文摘要:

Existing preference datasets for text-to-image models typically store only the final winner/loser images. This representation is insufficient for rectified flow (RF) models, whose generation is naturally indexed by a specific prior noise sample and follows a nearly straight denoising trajectory. In contrast, prior DPO-style alignment for diffusion models commonly estimates trajectories using an independent forward noising process, which can be mismatched to the true reverse dynamics and introduces unnecessary variance. We propose Prior Noise-Aware Preference Optimization (PNAPO), an off-policy alignment framework specialized for rectified flow. PNAPO augments preference data by retaining the paired prior noises used to generate each winner/loser image, turning the standard (prompt, winner, loser) triplet into a sextuple. Leveraging the straight-line property of RF, we estimate intermediate states via noise-image interpolation, which constrains the trajectory estimation space and yields a tighter surrogate objective for preference optimization. In addition, we introduce a dynamic regularization strategy that adapts the DPO regularization based on (i) the reward gap between winner and loser and (ii) training progress, improving stability and sample efficiency. Experiments on state-of-the-art RF T2I backbones show that PNAPO consistently improves preference metrics while substantially reducing training compute.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑