视频扩散模型中的早期故障检测与干预
Early Failure Detection and Intervention in Video Diffusion Models
AI总结:
本文提出了一种用于视频扩散模型的早期故障检测与干预方法,通过实时检查模块在生成过程中快速检测故障并提前干预,减少计算开销,提升生成质量。
摘要:
文本到视频(T2V)扩散模型已迅速发展,但生成仍偶尔失败,例如文本与视频对齐低或感知质量低。由于扩散采样是非确定性的,推理过程中难以确定生成是否会成功或失败,导致因试错重新生成而产生高计算成本。为此,我们提出了一种用于潜在T2V扩散模型的早期故障检测与诊断干预流程。对于检测,我们设计了一个实时检查(RI)模块,将潜在转换为中间视频预览,使可以利用已建立的文本-视频对齐评分器在RGB空间中进行检查。RI模块仅需39.2毫秒即可完成转换和检查过程。考虑到CogVideoX-5B在NVIDIA A100 GPU上生成480p、49帧视频时每个去噪步骤需要4.3秒,这非常高效。随后,我们仅在预测失败时触发分层和早退出干预流程。在CogVideoX-5B和Wan2.1-1.3B上的实验表明,与事后重新生成相比,VBench上的一致性提升可达2.64倍的时间开销减少。我们的方法也泛化到更高容量的设置,在Wan2.1-14B上保持有效,支持720p分辨率和81帧生成。此外,我们的流程是即插即用且与现有技术正交,显示出与提示细化和采样引导方法的无缝兼容性。我们还提供了证据表明,失败信号在去噪过程中早期出现,并且可以使用标准视觉-语言评估器在中间视频预览中检测到。
英文摘要:
Text-to-video (T2V) diffusion models have rapidly advanced, yet generations still occasionally fail in practice, such as low text-video alignment or low perceptual quality. Since diffusion sampling is non-deterministic, it is difficult to know during inference whether a generation will succeed or fail, incurring high computational cost due to trial-and-error regeneration. To address this, we propose an early failure detection and diagnostic intervention pipeline for latent T2V diffusion models. For detection, we design a Real-time Inspection (RI) module that converts latents into intermediate video previews, enabling the use of established text-video alignment scorers for inspection in the RGB space. The RI module completes the conversion and inspection process in just 39.2ms. This is highly efficient considering that CogVideoX-5B requires 4.3s per denoising step when generating a 480p, 49-frame video on an NVIDIA A100 GPU. Subsequently, we trigger a hierarchical and early-exit intervention pipeline only when failure is predicted. Experiments on CogVideoX-5B and Wan2.1-1.3B demonstrate consistency gains on VBench with up to 2.64 times less time overhead compared to post-hoc regeneration. Our method also generalizes to a higher-capacity setting, remaining effective on Wan2.1-14B with 720p resolution and 81-frame generation. Furthermore, our pipeline is plug-and-play and orthogonal to existing techniques, showing seamless compatibility with prompt refinement and sampling guidance methods. We also provide evidence that failure signals emerge early in the denoising process and are detectable within intermediate video previews using standard vision-language evaluators.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。