KVPO: ODE-Native GRPO用于通过KV语义探索实现自回归视频对齐
KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration
AI总结：
KVPO通过引入因果-语义探索范式和基于轨迹速度能量的替代策略，解决自回归视频生成器与人类偏好对齐的挑战，提升视频质量与文本对齐效果。
摘要：
对齐流式自回归（AR）视频生成器与人类偏好具有挑战性。现有强化学习方法主要依赖噪声探索和基于SDE的替代策略，这些策略与蒸馏AR模型的确定性ODE动态不匹配，倾向于扰动低层次外观而非高层次语义故事线进展，这对长周期一致性至关重要。为解决这些限制，我们提出了KVPO，一种用于对齐流式视频生成器的ODE原生在线组相对策略优化（GRPO）框架。对于多样性探索，KVPO引入了一种因果-语义探索范式，将变化源从随机噪声转移到历史KV缓存。通过随机路由历史KV条目，它构建了语义多样的生成分支，这些分支严格位于数据流形上。对于策略建模，KVPO引入了基于轨迹速度能量（TVE）的速度场替代策略，该策略在流匹配速度空间中量化分支可能性，并产生一个与原生ODE公式完全一致的奖励加权对比目标。在多个蒸馏AR视频生成器上的实验表明，在单提示短视频和多提示长视频设置中，视觉质量、运动质量和文本-视频对齐效果均取得一致提升。
英文摘要：
Aligning streaming autoregressive (AR) video generators with human preferences is challenging. Existing reinforcement learning methods predominantly rely on noise-based exploration and SDE-based surrogate policies that are mismatched to the deterministic ODE dynamics of distilled AR models, and tend to perturb low-level appearance rather than the high-level semantic storyline progression critical for long-horizon coherence. To address these limitations, we present KVPO, an ODE-native online Group Relative Policy Optimization (GRPO) framework for aligning streaming video generators. For diversity exploration, KVPO introduces a causal-semantic exploration paradigm that relocates the source of variation from stochastic noise to the historical KV cache. By stochastically routing historical KV entries, it constructs semantically diverse generation branches that remain strictly on the data manifold. For policy modeling, KVPO introduces a velocity-field surrogate policy based on Trajectory Velocity Energy (TVE), which quantifies branch likelihood in flow-matching velocity space and yields a reward-weighted contrastive objective fully consistent with the native ODE formulation. Experiments on multiple distilled AR video generators demonstrate consistent gains in visual quality, motion quality, and text-video alignment across both single-prompt short-video and multi-prompt long-video settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
