理解视频扩散模型中的注意力机制
Understanding Attention Mechanism in Video Diffusion Models
AI总结：
本文针对基于扩散的文本到视频合成模型中注意力机制的作用展开研究，通过信息论扰动分析揭示时空注意力图对视频质量的影响，并提出轻量级注意力矩阵操作方法以提升视频质量和实现文本引导编辑。
摘要：
文本到视频（T2V）合成模型，例如OpenAI的Sora，因其能够从文本提示生成高质量视频而备受关注。在基于扩散的T2V模型中，注意力机制是关键组件。然而，目前尚不清楚模型学习了哪些中间特征，以及T2V模型中的注意力块如何影响视频合成的各个方面，如图像质量和时间一致性。在本文中，我们使用信息论方法对T2V模型的空间和时间注意力块进行了深入的扰动分析。我们的结果表明，时间和空间注意力图不仅影响视频的时序和布局，还影响时空元素的复杂性以及合成视频的美学质量。值得注意的是，高熵注意力图通常是与卓越视频质量相关的关键元素，而低熵注意力图则与视频的帧内结构相关。基于我们的发现，我们提出了两种新颖的方法来提升视频质量并实现文本引导的视频编辑。这些方法完全依赖于对T2V模型中注意力矩阵的轻量级操作。我们的方法的有效性和高效性通过在多个数据集上的实验评估得到了进一步验证。
英文摘要：
Text-to-video (T2V) synthesis models, such as OpenAI's Sora, have garnered significant attention due to their ability to generate high-quality videos from a text prompt. In diffusion-based T2V models, the attention mechanism is a critical component. However, it remains unclear what intermediate features are learned and how attention blocks in T2V models affect various aspects of video synthesis, such as image quality and temporal consistency. In this paper, we conduct an in-depth perturbation analysis of the spatial and temporal attention blocks of T2V models using an information-theoretic approach. Our results indicate that temporal and spatial attention maps affect not only the timing and layout of the videos but also the complexity of spatiotemporal elements and the aesthetic quality of the synthesized videos. Notably, high-entropy attention maps are often key elements linked to superior video quality, whereas low-entropy attention maps are associated with the video's intra-frame structure. Based on our findings, we propose two novel methods to enhance video quality and enable text-guided video editing. These methods rely entirely on lightweight manipulation of the attention matrices in T2V models. The efficacy and effectiveness of our methods are further validated through experimental evaluation across multiple datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
