This paper identifies significant redundancy in the query-key interactions within self-attention mechanisms of diffusion transformer models, particularly during the early stages of denoising diffusion steps. In response to this observation, we present a novel diffusion transformer framework incorporating an additional set of mediator tokens to engage with queries and keys separately. By modulating the number of mediator tokens during the denoising generation phases, our model initiates the denoising process with a precise, non-ambiguous stage and gradually transitions to a phase enriched with detail. Concurrently, integrating mediator tokens simplifies the attention module's complexity to a linear scale, enhancing the efficiency of global attention processes. Additionally, we propose a time-step dynamic mediator token adjustment mechanism that further decreases the required computational FLOPs for generation, simultaneously facilitating the generation of high-quality images within the constraints of varied inference budgets. Extensive experiments demonstrate that the proposed method can improve the generated image quality while also reducing the inference cost of diffusion transformers. When integrated with the recent work SiT, our method achieves a state-of-the-art FID score of 2.01. The source code is available at https://github.com/LeapLabTHU/Attention-Mediators.
Patch-wise Mixed-Precision Quantization of Vision Transformer
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Depth-Wise Representation Development Under Blockwise Self-Supervised Learning for Video Vision Transformers
Heinrich Heine University, Düsseldorf, Germany(海因里希-海涅大学);Institute for Computational Visualistics, University of Koblenz, Germany(计算视觉学研究所);Institute of Neuroscience and Medicine (INM-1), Research Centre Jülich, Germany(神经科学与医学研究所);Helmholtz AI, Research Centre Jülich, Germany(海德堡人工智能研究所)
Weakly Supervised Intracranial Hemorrhage Segmentation using Head-Wise Gradient-Infused Self-Attention Maps from a Swin Transformer in Categorical Learning
ChWDTA: Channel-wise Wavelet-Domain Transformer Attention and Entropy Modeling for Learned Image Compression
Electrical and Computer Engineering Department, The University of British Columbia(英属哥伦比亚大学电气与计算机工程系);School of Engineering Science, Simon Fraser University(西蒙弗雷泽大学工程科学学院);School of Electronic Science and Technology, Eastern Institute of Technology(电子科学与技术学院,东部技术学院);Google LLC(谷歌公司)
Semantic Routing: Exploring Multi-Layer LLM Feature Weighting for Diffusion Transformers
Peking University(北京大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Nanjing University(南京大学);Fudan University(复旦大学)
Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Shenzhen Institute of Future Media Technology(深圳未来媒体技术研究院);Westlake University(西湖大学);ETH Zürich(苏黎世联邦理工学院)
Spach Transformer: Spatial and Channel-wise Transformer Based on Local and Global Self-attentions for PET Image Denoising
Gordon Center for Medical Imaging(戈登医学影像中心);Center for Advanced Medical Computing and Analysis(高级医学计算与分析中心);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);University of Florida(佛罗里达大学);University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心);Johns Hopkins University(约翰斯·霍普金斯大学)
针对 PET 图像噪声及三维全局自注意力计算成本高的问题,提出融合局部与全局 MSA 的空间通道编解码 Transformer,并在多示踪剂数据集上验证其去噪性能优于现有深度学习架构。
Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformers
School of Systems and Computing University of New South Wales(系统与计算学院 新南威尔士大学);School of Engineering Australian National University(工程学院 澳大利亚国立大学);School of Business University of New South Wales(商学院 新南威尔士大学);Australian Institute for Machine Learning University of Adelaide(机器学习研究所 阿德莱德大学);School of Computer Science and Engineering University of New South Wales(计算机科学与工程学院 新南威尔士大学);School of Computer Science University of Technology Sydney(计算机科学学院 技术大学悉尼)
AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers
Fraunhofer Heinrich-Hertz-Institute(弗劳恩霍夫海因茨·赫兹研究所);Technische Universität Berlin(柏林工业大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所(BIFOLD))
TranStable: Towards Robust Pixel-level Online Video Stabilization by Jointing Transformer and CNN
Dalian University of Technology(大连理工大学);Institute for Clarity in Documentation(文档清晰度研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康分局);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)
GroupedMixer: An Entropy Model with Group-wise Token-Mixers for Learned Image Compression
School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);School of Electronics Engineering and Computer Science, Peking University(北京大学电子工程与计算机科学学院)
BWCache: Accelerating Video Diffusion Transformers through Block-Wise Caching
School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China(人工智能学院,北京师范大学,北京);Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai 519087, China(人工智能与未来网络研究院,北京师范大学,珠海)
Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal
College of Intelligence and Computing(智能与计算学院);Tianjin University(天津大学);School of Cyberspace Science and Technology(网络空间安全学院);Beijing Institute of Technology(北京理工大学);National Key Laboratory of Science and Technology on Information System Security(信息系统安全科学与技术国家重点实验室)
UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer
Northeastern University(东北大学);Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University(东北大学医学影像智能计算教育部重点实验室);Amii, University of Alberta(阿尔伯塔大学阿尔伯塔机器智能研究所)
THAT: Token-wise High-frequency Augmentation Transformer for Hyperspectral Pansharpening
JPMorgan Chase(摩根大通公司);University of Missouri-Kansas City(密苏里大学-堪萨斯城分校);Ming Hsieh Department of Electrical and Computer Engineering(明希赫电气与计算机工程系);University of Southern California(南加州大学);Robinson Research Institute(罗宾逊研究学院);KTH Royal Institute of Technology(皇家理工学院);NEC Laboratories America(NEC美国实验室);University of Virginia(弗吉尼亚大学)
Xformer: Hybrid X-Shaped Transformer for Image Denoising
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);University of Sydney(悉尼大学);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);ETH Zürich(苏黎世联邦理工学院)
SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers
Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);University of Chinese Academy of Sciences(中国科学院大学);University of Extremadura(埃斯特雷马杜拉大学);Univ. Grenoble Alpes(格勒诺布尔阿尔卑斯大学);INRIA(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔理工学院)