arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.11146cs.CVcs.AI
收藏

超越基于VLM的奖励:扩散原生潜在奖励建模

Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modeling

The Hong Kong University of Science;Huawei Hong Kong AI Framework \& Data Technologies Lab;Tsinghua University;The Australian National University

Gongye Liu, Bo Yang, Yida Zhi, Zhizhou Zhong, Lei Ke, Didan Deng, Han Gao, Yongxiang Huang, Kaihao Zhang, Hongbo Fu, Wenhan Luo

AI总结:

提出扩散原生潜在奖励模型DiNa-LRM,直接在噪声扩散状态上进行偏好学习,通过噪声校准Thurstone似然和推理时噪声集成,实现高效且鲁棒的奖励建模。

摘要:

扩散和流匹配模型的偏好优化依赖于既具有判别鲁棒性又计算高效的奖励函数。视觉语言模型(VLM)凭借其丰富的多模态先验,已成为主要的奖励提供者,用于指导对齐。然而,它们的计算和内存成本可能很高,并且通过像素空间奖励优化潜在扩散生成器会引入域不匹配,使对齐复杂化。在本文中,我们提出DiNa-LRM,一种扩散原生潜在奖励模型,直接在噪声扩散状态上制定偏好学习。我们的方法引入了一种噪声校准的Thurstone似然,具有扩散噪声依赖的不确定性。DiNa-LRM利用预训练的潜在扩散骨干网络,配备时间步条件奖励头,并支持推理时噪声集成,提供了一种扩散原生的机制用于测试时缩放和鲁棒奖励。在图像对齐基准测试中,DiNa-LRM显著优于现有的基于扩散的奖励基线,并以一小部分计算成本实现了与最先进VLM竞争的性能。在偏好优化中,我们证明DiNa-LRM改善了偏好优化动态,实现了更快且更资源高效的模型对齐。

英文摘要:

Preference optimization for diffusion and flow-matching models relies on reward functions that are both discriminatively robust and computationally efficient. Vision-Language Models (VLMs) have emerged as the primary reward provider, leveraging their rich multimodal priors to guide alignment. However, their computation and memory cost can be substantial, and optimizing a latent diffusion generator through a pixel-space reward introduces a domain mismatch that complicates alignment. In this paper, we propose DiNa-LRM, a diffusion-native latent reward model that formulates preference learning directly on noisy diffusion states. Our method introduces a noise-calibrated Thurstone likelihood with diffusion-noise-dependent uncertainty. DiNa-LRM leverages a pretrained latent diffusion backbone with a timestep-conditioned reward head, and supports inference-time noise ensembling, providing a diffusion-native mechanism for test-time scaling and robust rewarding. Across image alignment benchmarks, DiNa-LRM substantially outperforms existing diffusion-based reward baselines and achieves performance competitive with state-of-the-art VLMs at a fraction of the computational cost. In preference optimization, we demonstrate that DiNa-LRM improves preference optimization dynamics, enabling faster and more resource-efficient model alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑