arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.01609cs.CV
收藏

令牌剪枝用于扩散变换器中的上下文生成

Token Pruning for In-Context Generation in Diffusion Transformers

University of Science and Technology of China, Anhui, China(中国科学技术大学);Beihang University, Beijing, China(北京航空航天大学);Tencent PCG, China(腾讯PCG)

Junqing Lin, Xingyu Zheng, Pei Cheng, Bin Fu, Jingwei Sun, Guangzhong Sun

AI总结:

ToPi通过无训练的令牌剪枝框架提升扩散变换器的上下文生成效率,实现30%以上的推理加速同时保持生成质量。

摘要:

上下文生成通过参考示例增强了扩散变换器(DiTs),从而实现可控的图像到图像生成。然而,由此产生的输入连接显著增加了序列长度,造成显著的计算瓶颈。现有的令牌减少技术主要针对文本到图像合成,但在这种范式中效果不佳,因为它们应用统一的减少策略,忽略了参考上下文和目标潜在之间在空间、时间和功能维度上的固有不对称性。为弥合这一差距,我们引入ToPi,一种针对DiTs中上下文生成的无训练令牌剪枝框架。具体而言,ToPi利用离线校准驱动的敏感性分析来识别关键的注意力层,作为冗余估计的稳健代理。利用这些层,我们推导出一种新的影响度量来量化每个上下文令牌的贡献,以实现选择性剪枝,并结合一种时间更新策略,以适应扩散轨迹的演变。实证评估表明,ToPi可以在保持复杂图像生成任务的结构保真度和视觉一致性的同时,实现超过30%的推理加速。

英文摘要:

In-context generation significantly enhances Diffusion Transformers (DiTs) by enabling controllable image-to-image generation through reference examples. However, the resulting input concatenation drastically increases sequence length, creating a substantial computational bottleneck. Existing token reduction techniques, primarily tailored for text-to-image synthesis, fall short in this paradigm as they apply uniform reduction strategies, overlooking the inherent role asymmetry between reference contexts and target latents across spatial, temporal, and functional dimensions. To bridge this gap, we introduce ToPi, a training-free token pruning framework tailored for in-context generation in DiTs. Specifically, ToPi utilizes offline calibration-driven sensitivity analysis to identify pivotal attention layers, serving as a robust proxy for redundancy estimation. Leveraging these layers, we derive a novel influence metric to quantify the contribution of each context token for selective pruning, coupled with a temporal update strategy that adapts to the evolving diffusion trajectory. Empirical evaluations demonstrate that ToPi can achieve over 30\% speedup in inference while maintaining structural fidelity and visual consistency across complex image generation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑