arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.23903cs.CV
收藏

Geo-Align: 通过度量几何奖励实现视频生成对齐

Geo-Align: Video Generation Alignment via Metric Geometry Reward

USTC(中国科学技术大学);Shanghai AI Lab(上海人工智能实验室);ZJU(浙江大学)

Zizun Li, Haoyu Guo, Runzhe Teng, Chunhua Shen, Tong He

AI总结:

提出首个基于强化学习的框架Geo-Align,通过度量3D估计器提取相机轨迹并惩罚偏差,利用真实条件视频和合成目标轨迹消除配对数据依赖,提升相机控制精度和视觉保真度。

摘要:

近年来,相机控制的视频生成取得了显著进展。然而,现有的视频到视频重渲染方法主要依赖于使用合成数据集的监督微调。目前,同步的多视角真实世界视频数据极度稀缺。因此,当前范式在处理分布外的真实世界视频时通常表现出有限的泛化能力,模型难以准确遵循物理尺度和相机轨迹。为了弥补这一差距,我们提出了Geo-Align,这是第一个专门为相机控制视频重渲染设计的强化学习框架。基于预训练模型,我们通过尺度感知的感知奖励机制优化模型。具体来说,我们引入了一个度量3D估计器,从生成的视频中提取精确的相机轨迹,明确惩罚旋转和平移的偏差。此外,我们精心设计了一种基于真实条件视频和从合成数据导出的目标相机轨迹的数据流水线策略,消除了对配对数据的依赖。大量实验表明,Geo-Align在精确的相机可控性和视觉保真度方面始终优于现有的监督学习基线,证明了我们方法的有效性。

英文摘要:

Camera-controlled video generation has achieved remarkable progress in recent years. However, existing video-to-video re-rendering methods primarily rely on Supervised Fine-Tuning using synthetic datasets. At present, there is an extreme scarcity of synchronized, multi-view real-world video data. Consequently, the prevailing paradigm often exhibits limited generalization when processing out-of-distribution real-world videos, with models struggling to accurately adhere to physical scales and camera trajectories. To bridge this gap, we propose Geo-Align, the first Reinforcement Learning framework specifically designed for camera-controlled video re-rendering. Built upon a pretrained model, we optimize the model through a scale-aware perceptual reward mechanism. Specifically, we introduce a metric 3D estimator to extract precise camera trajectories from generated videos, explicitly penalizing deviations in rotation and translation. Furthermore, we meticulously designed a data pipeline strategy based on real-world conditioning videos and target camera trajectories derived from synthetic data, eliminating the reliance on paired data. Extensive experiments demonstrate that Geo-Align consistently outperforms existing supervised learning baselines in both precise camera controllability and visual fidelity, indicating the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑