Kaleido：通过利用潜在空间相关性对视频扩散变压器进行算法-硬件协同设计

中文摘要 AI 辅助

视频扩散变压器（vDiTs）能生成高质量视频，但由于扩散时间步长和自注意力计算，计算成本极高。随着扩散时间步长减少，自注意力计算成本成为主要瓶颈。现有加速方法大多继承大语言模型的稀疏注意力技术，未考虑视频数据独特的时空相关性。本文提出Kaleido，一种算法-硬件协同设计，通过利用潜在空间中的通道级时空相关性加速vDiTs中的所有操作。基于此，提出轻量级通道级重用算法，在保留比现有方法更高生成质量（>17dB）的同时跳过冗余计算。还设计了具有可重构处理元件的脉动阵列加速器和轻量级数据调度器。对三个主流vDiT模型的评估表明，Kaleido比现有加速器加速高达5.9倍，节能16.0倍。

英文摘要

Video diffusion transformers (vDiTs) generate high quality video but introduce extremely high compute cost due to the long diffusion timesteps and self attention computation. As diffusion timesteps are reduced, the computation cost of self attention becomes the dominant bottleneck. Existing acceleration approaches largely inherit sparse attention techniques from large language models, which fail to consider the unique spatiotemporal correlation of video data. This paper presents Kaleido, an algorithm hardware codesign that accelerates all operations in vDiTs by exploiting channel-wise spatiotemporal correlations in latent space. Based on this insight, we propose a lightweight channelwise reuse algorithm that skips redundant computations by reusing partial results while preserving higher generative quality than prior methods (>17 dB). To efficiently support this algorithm, we design a systolic array like accelerator with reconfigurable processing elements and a lightweight data dispatcher to mitigate irregular sparsity and data access patterns introduced by our reuse algorithm. Evaluations across three mainstream vDiT models show that Kaleido achieves up to 5.9x speedup and 16.0x energy savings over state of the art accelerators.

发表机构

• Shanghai Jiao Tong University（上海交通大学）
• Shanghai Jiao Tong University, Shanghai Qi Zhi Institute（上海交通大学、上海颀智研究所）
• Huawei Technologies（华为技术有限公司）
• ICT, Chinese Academy of Sciences（信息科技研究所、中国科学院）

机构由 AI 辅助整理，请以论文原文为准。