MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）；vivo Mobile Communication Co., Ltd.（维沃移动通信有限公司）
Classifier-Free Guidance (CFG) is a widely used inference-time technique to boost the image quality of diffusion models. Yet, its reliance on text conditions prevents its use in unconditional generation. We propose a simple method to enable CFG-like guidance for both conditional and unconditional generation. The key idea is to generate a perturbed prediction via simple token swap operations, and use the direction between it and the clean prediction to steer sampling towards higher-fidelity distributions. In practice, we swap pairs of most semantically dissimilar token latents in either spatial or channel dimensions. Unlike existing methods that apply perturbation in a global or less constrained manner, our approach selectively exchanges and recomposes token latents, allowing finer control over perturbation and its influence on generated samples. Experiments on MS-COCO 2014, MS-COCO 2017, and ImageNet datasets demonstrate that the proposed Self-Swap Guidance (SSG), when applied to popular diffusion models, outperforms previous condition-free methods in image fidelity and prompt alignment under different set-ups. Its fine-grained perturbation granularity also improves robustness, reducing side-effects across a wider range of perturbation strengths. Overall, SSG extends CFG to a broader scope of applications including both conditional and unconditional generation, and can be readily inserted into any diffusion model as a plug-in to gain immediate improvements.
Deep neural networks often make decisions based on the spurious correlations inherent in the dataset, failing to generalize in an unbiased data distribution. Although previous approaches pre-define the type of dataset bias to prevent the network from learning it, recognizing the bias type in the real dataset is often prohibitive. This paper proposes a novel bias-tailored augmentation-based approach, BiaSwap, for learning debiased representation without requiring supervision on the bias type. Assuming that the bias corresponds to the easy-to-learn attributes, we sort the training images based o…
Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space
The University of British Columbia（不列颠哥伦比亚大学）；The Ohio State University（俄亥俄州立大学）；Czech Technical University in Prague（布拉格捷克理工大学）；TerraSense Analytics（TerraSense Analytics公司）
Guiding What Not to Generate: Automated Negative Prompting for Text-Image Alignment
Department of Electrical and Computer Engineering, Seoul National University（电子与计算机工程系，首尔国立大学）；Amazon（亚马逊）；IPAI, AIIS, ASRI, INMC, and ISRC, Seoul National University（IPAI、AIIS、ASRI、INMC 和 ISRC，首尔国立大学）
FlowCoMotion: Text-to-Motion Generation via Token-Latent Flow Modeling
School of Artificial Intelligence and Data Science, University of Science and Technology of China（人工智能与数据科学学院，中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China（苏州先进研究院，中国科学技术大学）
Disrupting Diffusion: Token-Level Attention Erasure Attack against Diffusion-based Customization
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；School of Computer Science and Tech., University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Information Engineering, CAS（信息工程研究所，中国科学院）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）
The application of zero-shot learning in computer vision has been revolutionized by the use of image-text matching models. The most notable example, CLIP, has been widely used for both zero-shot classification and guiding generative models with a text prompt. However, the zero-shot use of CLIP is unstable with respect to the phrasing of the input text, making it necessary to carefully engineer the prompts used. We find that this instability stems from a selective similarity score, which is based only on a subset of the semantically meaningful input tokens. To mitigate it, we present a novel ex…
