滚动汇:在有限时间训练和开放-ended测试之间架桥
Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusion
AI总结:
本文提出Rolling Sink,通过分析AR缓存维护,实现无需训练的超长视频生成,提升视觉质量和时间一致性。
摘要:
最近,自回归(AR)视频扩散模型取得了显著成效。然而,由于其训练时间有限,当在更长的时域测试时,会出现训练-测试差距,导致快速的视觉退化。受Self Forcing启发,本文研究训练时间之外的训练-测试差距,即训练时有限时域与测试时开放-ended时域之间的差距。由于开放-ended测试可超越任何有限训练窗口,且长视频训练计算成本高,我们追求一种无需训练的解决方案。为了探索无需训练的解决方案,我们系统分析了AR缓存维护。这些见解导致了Rolling Sink。基于Self Forcing(仅训练5秒片段),Rolling Sink在测试时有效扩展AR视频合成至超长时域(例如,5-30分钟,16 FPS),具有一致的主体、稳定的色彩、连贯的结构和流畅的运动。通过广泛实验表明,Rolling Sink在超长时域的视觉保真度和时间一致性方面优于现有最佳基线。项目页面:https://rolling-sink.github.io/
英文摘要:
Autoregressive (AR) video diffusion models in the Self Forcing family generate a video chunk by chunk, each conditioned on the ones before it, so they can keep rolling out with no preset length. In practice they are trained on short, fixed clips (e.g., 5s). Once a rollout passes this training horizon, the cache fills with self-generated chunks the model never saw in training, and quality degrades: colors oversaturate, identities drift, structures collapse, and motion fades. We frame this failure as a cache-behavior mismatch, a domain shift between the within-horizon caches seen in training and those produced by open-ended testing, and address it with Rolling Sink, a training-free inference-time cache policy. Within the base model's fixed cache budget, Rolling Sink holds the cache close to its stable within-horizon behavior: it restores low-drift content, sliding temporal indices, and evolving semantics, and keeps a small recent context. Attention sink and temporal re-indexing are existing mechanisms. Our contribution is the mismatch formulation, the joint bounded-cache policy, and the Rolling Semantics schedule. On two 5s-trained bases, Self Forcing and Causal Forcing, Rolling Sink attains the best averaged rank on VBench-Long at both 1- and 5-minute rollouts under the same inference setting, at about 0.56% throughput change, and holds GPU memory constant. It also sustains a 30-minute qualitative rollout, 360 times its training horizon. We validate at 1- and 5-minute rollouts and do not claim guaranteed quality at arbitrary duration. Project page: https://rolling-sink.github.io/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。