arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.25527cs.CV
收藏

超越黄金数据:通过时间步选择性训练解决运动-视觉质量困境

Beyond the Golden Data: Resolving the Motion-Vision Quality Dilemma via Timestep Selective Training

Tsinghua University(清华大学);Kling Team, Kuaishou Technology(快手科技 Kling 团队)

Xiangyang Luo, Qingyu Li, Yuming Li, Guanbo Huang, Yongjie Zhu, Wenyu Qin, Meng Wang, Pengfei Wan, Shao-Lun Huang

AI总结:

本文提出TQD方法,通过时间步选择性训练解决视频生成中运动与视觉质量的矛盾,证明在不平衡数据上训练可超越传统高质量数据训练效果。

摘要:

最近视频生成模型的进展取得了显著成果。然而,这些模型严重依赖高质量数据,即同时具有高视觉质量和高运动质量。本文识别了视频数据整理中的关键挑战:运动-视觉质量困境。我们发现视觉质量和运动强度本质上存在负相关,使得难以获得在两者方面都卓越的黄金数据。为解决这一挑战,我们首先研究了视频扩散模型的层次学习动态,并对质量退化的样本进行基于梯度的分析。我们发现质量不平衡的数据可以在适当的时间步产生与黄金数据相似的梯度。基于此,我们引入了训练过程中的时间步选择新概念。我们提出时间步感知质量解耦(TQD),通过修改数据采样分布以更好地匹配模型的学习过程。对于某些类型的数据,采样分布倾向于运动丰富的数据向较高时间步,而高视觉质量数据更可能在较低时间步被采样。通过广泛的实验,我们证明TQD能够仅在分离的不平衡数据上训练,实现超越传统训练使用更好数据的性能,挑战了视频生成中完美数据的必要性。此外,我们的方法在训练高质量数据时也提升了模型性能,展示了其在不同数据场景中的有效性。

英文摘要:

Recent advances in video generation models have achieved impressive results. However, these models heavily rely on the use of high-quality data that combines both high visual quality and high motion quality. In this paper, we identify a key challenge in video data curation: the Motion-Vision Quality Dilemma. We discovered that visual quality and motion intensity inherently exhibit a negative correlation, making it hard to obtain golden data that excels in both aspects. To address this challenge, we first examine the hierarchical learning dynamics of video diffusion models and conduct gradient-based analysis on quality-degraded samples. We discover that quality-imbalanced data can produce gradients similar to golden data at appropriate timesteps. Based on this, we introduce the novel concept of Timestep selection in Training Process. We propose Timestep-aware Quality Decoupling (TQD), which modifies the data sampling distribution to better match the model's learning process. For certain types of data, the sampling distribution is skewed toward higher timesteps for motion-rich data, while high visual quality data is more likely to be sampled during lower timesteps. Through extensive experiments, we demonstrate that TQD enables training exclusively on separated imbalanced data to achieve performance surpassing conventional training with better data, challenging the necessity of perfect data in video generation. Moreover, our method also boosts model performance when trained on high-quality data, showcasing its effectiveness across different data scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑