Qihoo-T2X：一种用于文本到任意任务的高效代理令牌化扩散Transformer
Qihoo-T2X: An Efficient Proxy-Tokenized Diffusion Transformer for Text-to-Any-Task
AI总结：
本文提出代理令牌化扩散Transformer（PT-DiT），通过稀疏代表性令牌注意力结合窗口与移位窗口注意力，在保持图像和视频生成性能的同时显著降低计算复杂度，并基于此构建了Qihoo-T2X系列模型。
摘要：
扩散Transformer中的全局自注意力机制由于视觉信息的稀疏性和冗余性而存在冗余计算，并且空间窗口内令牌的注意力图表现出显著相似性。为了解决这种冗余，我们提出了代理令牌化扩散Transformer（PT-DiT），它采用稀疏代表性令牌注意力（其中代表性令牌的数量远小于令牌总数）来高效地建模全局视觉信息。具体来说，在每个Transformer块中，我们从每个时空窗口计算一个平均令牌，作为该区域的代理令牌。全局语义通过代理令牌的自注意力被捕获，然后通过交叉注意力注入所有潜在令牌。同时，我们引入窗口注意力和移位窗口注意力，以解决稀疏注意力机制在细节建模方面的局限性。在精心设计的PT-DiT基础上，我们进一步开发了Qihoo-T2X系列，其中包括用于T2I、T2V和T2MV任务的多种模型。实验结果表明，PT-DiT在图像和视频生成任务中实现了具有竞争力的性能，同时降低了计算复杂度（例如，与DiT相比降低了49%，与PixArt-α相比降低了34%）。Qihoo-T2X的视觉展示和源代码可在 https://360cvgroup.github.io/Qihoo-T2X/ 获取。
英文摘要：
The global self-attention mechanism in diffusion transformers involves redundant computation due to the sparse and redundant nature of visual information, and the attention map of tokens within a spatial window shows significant similarity. To address this redundancy, we propose the Proxy-Tokenized Diffusion Transformer (PT-DiT), which employs sparse representative token attention (where the number of representative tokens is much smaller than the total number of tokens) to model global visual information efficiently. Specifically, within each transformer block, we compute an averaging token from each spatial-temporal window to serve as a proxy token for that region. The global semantics are captured through the self-attention of these proxy tokens and then injected into all latent tokens via cross-attention. Simultaneously, we introduce window and shift window attention to address the limitations in detail modeling caused by the sparse attention mechanism. Building on the well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a variety of models for T2I, T2V, and T2MV tasks. Experimental results show that PT-DiT achieves competitive performance while reducing the computational complexity in both image and video generation tasks (e.g., a 49% reduction compared to DiT and a 34% reduction compared to PixArt-$α$). The visual exhibition and source code of Qihoo-T2X is available at https://360cvgroup.github.io/Qihoo-T2X/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
