扩散变换器（DiTs）在图像生成质量上表现出色，但其计算复杂度与token数量呈二次关系。尽管已提出多种token减少（TR）方法以缓解这一成本，但它们忽略了生成模型的主要目标：最小化恢复误差，这需要反映输出token的相似性。它们仅依赖于输入token相似性，这是来自仅减少的ViT范式继承的，导致与该目标的根本不一致。为弥合这一差距，我们提出DiTo，一种新的TR范式，其重点转向以输出为中心的token减少。基于观察到输出token相似性在相邻时间步中保持一致，DiTo利用先前步骤的相似性作为有效代理，在匹配时间步中建立token对应关系，然后在多个后续减少时间步中重用。为了优化这种交错调度，我们提出Pair Match Ratio（PMR）引导的区间调度，以确定最佳匹配频率。此外，为了减轻由重复重用导致的局部近似误差和由此产生的阻塞伪影，我们提出频率感知的token匹配，通过引入选择频率惩罚。广泛的实验表明，DiTo在可比的加速下，比现有TR方法在PSNR上高出1.6-3.9 dB，实现了更优的帕累托前沿。
英文摘要：
Diffusion Transformers (DiTs) achieve superior image generation quality but suffer from quadratic computational complexity relative to token count. While various token reduction (TR) methods have been proposed to mitigate this cost, they overlook the primary objective of generative models: minimizing recovery error, which requires reflecting output token similarity. They rely solely on input token similarity inherited from reduction-only ViT paradigms, leading to a fundamental misalignment with this objective.
To bridge this gap, we propose DiTo, a novel TR paradigm that shifts the focus toward output-centric token reduction. Based on the observation that output token similarity is consistently preserved across adjacent timesteps, DiTo utilizes prior-step similarities as an effective proxy to establish token correspondences at a Matching timestep, which are then reused across multiple subsequent Reduction timesteps. To optimize this interleaved scheduling, we propose Pair Match Ratio (PMR)-guided Interval Scheduling to determine the optimal matching frequency. Furthermore, to mitigate localized approximation errors and resulting blocking artifacts caused by repeated reuse, we propose Frequency-aware Token Matching by incorporating a selection-frequency penalty. Extensive experiments demonstrate that DiTo consistently outperforms existing TR methods with 1.6-3.9 dB higher PSNR at comparable speedups, achieving a superior Pareto frontier.
University of Science and Technology of China（中国科学技术大学）；Microsoft Cloud AI（微软云人工智能）；Snap Inc.（Snap公司）；City University of Hong Kong（香港城市大学）；University of Southern California（南加利福尼亚大学）
Spatially-adaptive normalization is remarkably successful recently in conditional semantic image synthesis, which modulates the normalized activation with spatially-varying transformations learned from semantic layouts, to preserve the semantic information from being washed away. Despite its impressive performance, a more thorough understanding of the true advantages inside the box is still highly demanded, to help reduce the significant computation and parameter overheads introduced by these new structures. In this paper, from a return-on-investment point of view, we present a deep analysis o…
Learning a good representation for space-time correspondence is the key for various computer vision tasks, including tracking object bounding boxes and performing video object pixel segmentation. To learn generalizable representation for correspondence in large-scale, a variety of self-supervised pretext tasks are proposed to explicitly perform object-level or patch-level similarity learning. Instead of following the previous literature, we propose to learn correspondence using Video Frame-level Similarity (VFS) learning, i.e, simply learning from comparing video frames. Our work is inspired b…
HAPS: Rethinking Image Similarity for Virtual Staining
Skolkovo Institute of Science and Technology（斯克洛洛沃科学与技术研究所）；University of Sharjah（沙迦大学）；National Medical Research Radiological Centre of the Ministry of Health of the Russian Federation（俄罗斯联邦卫生部国家医学研究放射中心）
Surrey Institute for People-Centred AI（苏格兰人本人工智能研究所）；Centre For Vision, Speech and Signal Processing (CVSSP)（视觉、语音和信号处理中心）；Jiangnan University（江南大学）；Carnegie Mellon University（卡内基梅隆大学）
Rethinking Evaluation Metrics of Open-Vocabulary Segmentaion
Harbin Engineering University（哈尔滨工程大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；The University of California, Merced（加州大学默塞德分校）
Boston University（波士顿大学）；Kuaishou Technology（快手科技）
Most existing methods in vision-language retrieval match two modalities by either comparing their global feature vectors which misses sufficient information and lacks interpretability, detecting objects in images or videos and aligning the text with fine-grained features which relies on complicated model designs, or modeling fine-grained interaction via cross-attention upon visual and textual tokens which suffers from inferior efficiency. To address these limitations, some recent works simply aggregate the token-wise similarities to achieve fine-grained alignment, but they lack intuitive expla…
Rethinking Representations for Cross-Domain Infrared Small Target Detection: A Generalizable Perspective from the Frequency Domain
Department of Mathematics, Hong Kong Baptist University（香港浸会大学数学系）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
Simon Fraser University（西蒙弗雷泽大学）；Adobe Research（Adobe研究院）
We present a deep neural network to predict structural similarity between 2D layouts by leveraging Graph Matching Networks (GMN). Our network, coined LayoutGMN, learns the layout metric via neural graph matching, using an attention-based GMN designed under a triplet network setting. To train our network, we utilize weak labels obtained by pixel-wise Intersection-over-Union (IoUs) to define the triplet loss. Importantly, LayoutGMN is built with a structural bias which can effectively compensate for the lack of structure awareness in IoUs. We demonstrate this on two prominent forms of layouts, v…
Rethinking Metrics and Diffusion Architecture for 3D Point Cloud Generation
Mines Paris, Université PSL（Mines Paris，Université PSL）；Centre des Matériaux (MAT), UMR7633 CNRS（Centre des Matériaux（MAT），UMR7633 CNRS）；Centre de Morphologie Mathématique (CMM)（Centre de Morphologie Mathématique（CMM））；Centre de Mise en Forme des Matériaux (CEMEF), UMR7635 CNRS（Centre de Mise en Forme des Matériaux（CEMEF），UMR7635 CNRS）
本文针对三维点云生成评估指标鲁棒性不足的问题，提出样本对齐、DCD及新指标SNC，并设计Diffusion Point Transformer架构，在ShapeNet上取得领先性能。
