EchoCache：面向高效音频驱动视频生成的能量引导跨模态缓存

中文摘要 AI 辅助

音频驱动视频生成（A2V）在合成时间连贯、音视对齐的视频方面已取得良好进展，但由于扩散模型的迭代去噪过程，其推理成本仍较高。现有缓存方法主要利用视觉特征中的时间冗余，却忽略了A2V的跨模态对齐——音频驱动视觉生成时具有高度非均匀的时间重要性。本文指出现有A2V缓存方法存在时间-语义、计算-存储两个层面的错位问题。为解决这些问题，我们提出EchoCache，这是一种用于高效A2V生成的能量引导跨模态缓存框架。EchoCache利用音频时频能量作为显著性锚点，引导隐层级缓存更新，还引入了带量化缓存管理的动态时间步长-隐层缓存机制，以实现效率与内存的联合优化。在主流A2V模型上开展的大量实验表明，EchoCache在保持生成质量和音视一致性的同时，持续优化了延迟-质量权衡；尤其在EMTD基准上针对Wan2.2-S2V模型，EchoCache实现了2.46倍的加速，且整体性能最优。代码可在该https URL获取。

英文摘要

Audio-driven video generation (A2V) has achieved promising progress in synthesizing temporally coherent and audio-visually aligned videos, yet its inference remains expensive due to the iterative denoising process of diffusion models. Existing caching methods mainly exploit temporal redundancy in visual features while overlooking the cross-modal alignment of A2V, where audio drives visual generation with highly non-uniform temporal importance. In this paper, we identify two levels of misalignment in existing A2V caching methods: temporal-semantic and computation-storage misalignment. To address them, we propose EchoCache, an energy-guided cross-modal caching framework for efficient A2V generation. EchoCache leverages audio time-frequency energy as a saliency anchor to guide latent-level cache updates and further introduces a dynamic timestep-latent caching mechanism with quantized cache management for joint efficiency and memory optimization. Extensive experiments on mainstream A2V models show that EchoCache consistently improves the latency-quality trade-off while preserving generation quality and audio-visual consistency. In particular, on Wan2.2-S2V over the EMTD benchmark, EchoCache achieves a 2.46x speedup with the best overall performance. Code is available at https://github.com/IF-LAB-PKU/EchoCache.

发表机构

• Peking University（北京大学）
• Taiyuan University of Technology（太原理工大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

EchoCache is honored to be accepted by ACM MM 2026