通过异构缓存加速基于扩散的视频编辑：超越采样去噪时间步的全计算
Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep
AI总结：
本文提出HetCache框架，通过利用扩散基于掩码视频到视频生成中的异构性，减少冗余注意力运算，提升视频编辑效率与质量。
摘要：
基于扩散的视频编辑已发展为高质量和灵活内容生成的重要范式。然而，尽管其通用性和强大的建模能力，扩散变换器（DiT）仍因迭代去噪过程而计算成本高，给实际部署带来挑战。现有视频扩散加速方法主要利用去噪时间步级的特征重用，减轻去噪过程的冗余，但忽略了DiT内部的架构冗余，即许多空间时间令牌上的注意力操作被冗余执行，对模型输出贡献有限。本文引入HetCache，一种无需训练的扩散加速框架，旨在利用扩散基于掩码视频到视频（MV2V）生成和编辑中的内在异构性。不同于均匀重用或随机采样令牌，HetCache评估各类令牌在指定计算步骤中的上下文相关性和交互强度。受空间先验引导，它将DiT模型中的空间时间令牌分为上下文令牌和生成令牌，并选择性地缓存与生成令牌具有最强相关性和最代表性的上下文令牌。该策略减少冗余注意力运算，同时保持编辑的一致性和保真度。实验表明，HetCache在常用基础模型上实现了显著的加速，包括2.67倍的延迟提速和FLOPs减少，且对编辑质量的降级可忽略。
英文摘要：
Diffusion-based video editing has emerged as an important paradigm for high-quality and flexible content generation. However, despite their generality and strong modeling capacity, Diffusion Transformers (DiT) remain computationally expensive due to the iterative denoising process, posing challenges for practical deployment. Existing video diffusion acceleration methods primarily exploit denoising timestep-level feature reuse, which mitigates the redundancy in denoising process, but overlooks the architectural redundancy within the DiT that many attention operations over spatio-temporal tokens are redundantly executed, offering little to no incremental contribution to the model output. This work introduces HetCache, a training-free diffusion acceleration framework designed to exploit the inherent heterogeneity in diffusion-based masked video-to-video (MV2V) generation and editing. Instead of uniformly reuse or randomly sampling tokens, HetCache assesses the contextual relevance and interaction strength among various types of tokens in designated computing steps. Guided by spatial priors, it divides the spatial-temporal tokens in DiT model into context and generative tokens, and selectively caches the context tokens that exhibit the strongest correlation and most representative semantics with generative ones. This strategy reduces redundant attention operations while maintaining editing consistency and fidelity. Experiments show that HetCache achieves a noticeable acceleration, including a 2.67$\times$ latency speedup and FLOPs reduction over commonly used foundation models, with negligible degradation in editing quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
