AR-CoPO: 利用对比策略优化对齐自回归视频生成
AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization
AI总结：
提出AR-CoPO框架，通过分叉机制构建邻域候选、分块级对齐和半在线训练策略，解决流式自回归视频生成中RLHF对齐难题，提升跨域泛化与人类偏好对齐。
摘要：
流式自回归（AR）视频生成器结合少步蒸馏实现了低延迟、高质量的合成，但仍难以通过人类反馈强化学习（RLHF）进行对齐。现有的基于SDE的GRPO方法在此设置下面临挑战：少步ODE和一致性模型采样器偏离标准流匹配ODE，且其短、低随机性的轨迹对初始化噪声高度敏感，使得中间SDE探索无效。我们提出AR-CoPO（自回归对比策略优化），一个将Neighbor GRPO对比视角适应于流式AR生成的框架。AR-CoPO通过分叉机制引入分块级对齐，该机制在随机选择的分块上构建邻域候选，分配序列级奖励，并执行局部GRPO更新。我们进一步提出一种半在线训练策略，通过利用参考展开的重放缓冲区进行开发来补充在线探索，从而提升跨域生成质量。在Self-Forcing上的实验表明，AR-CoPO在跨域泛化和域内人类偏好对齐方面均优于基线，提供了真正对齐而非奖励破解的证据。
英文摘要：
Streaming autoregressive (AR) video generators combined with few-step distillation achieve low-latency, high-quality synthesis, yet remain difficult to align via reinforcement learning from human feedback (RLHF). Existing SDE-based GRPO methods face challenges in this setting: few-step ODEs and consistency model samplers deviate from standard flow-matching ODEs, and their short, low-stochasticity trajectories are highly sensitive to initialization noise, rendering intermediate SDE exploration ineffective. We propose AR-CoPO (AutoRegressive Contrastive Policy Optimization), a framework that adapts the Neighbor GRPO contrastive perspective to streaming AR generation. AR-CoPO introduces chunk-level alignment via a forking mechanism that constructs neighborhood candidates at a randomly selected chunk, assigns sequence-level rewards, and performs localized GRPO updates. We further propose a semi-on-policy training strategy that complements on-policy exploration with exploitation over a replay buffer of reference rollouts, improving generation quality across domains. Experiments on Self-Forcing demonstrate that AR-CoPO improves both out-of-domain generalization and in-domain human preference alignment over the baseline, providing evidence of genuine alignment rather than reward hacking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
