Though the background is an important signal for image classification, over reliance on it can lead to incorrect predictions when spurious correlations between foreground and background are broken at test time. Training on a dataset where these correlations are unbiased would lead to more robust models. In this paper, we propose such a dataset called Diffusion Dreamed Distribution Shifts (D3S). D3S consists of synthetic images generated through StableDiffusion using text prompts and image guides obtained by pasting a sample foreground image onto a background template image. Using this scalable approach we generate 120K images of objects from all 1000 ImageNet classes in 10 diverse backgrounds. Due to the incredible photorealism of the diffusion model, our images are much closer to natural images than previous synthetic datasets. D3S contains a validation set of more than 17K images whose labels are human-verified in an MTurk study. Using the validation set, we evaluate several popular DNN image classifiers and find that the classification performance of models generally suffers on our background diverse images. Next, we leverage the foreground & background labels in D3S to learn a foreground (background) representation that is invariant to changes in background (foreground) by penalizing the mutual information between the foreground (background) features and the background (foreground) labels. Linear classifiers trained on these features to predict foreground (background) from foreground (background) have high accuracies at 82.9% (93.8%), while classifiers that predict these labels from background and foreground have a much lower accuracy of 2.4% and 45.6% respectively. This suggests that our foreground and background features are well disentangled. We further test the efficacy of these representations by training classifiers on a task with strong spurious correlations.
Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniques
Institute of Information Engineering, CAS(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Institute of Computing Technology, CAS(中国科学院计算技术研究所);Peng Cheng Laboratory(鹏城实验室);School of Computer Science and Tech., University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);School of Cyber Science and Tech., Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区网络空间科学与技术学院);Key Laboratory of Big Data Mining and Knowledge Management, CAS(中国科学院大数据挖掘与知识管理重点实验室)
Adaptive Domain Shift in Diffusion Models for Cross-Modality Image Translation
Laplace Lab at University of Tennessee(田纳西大学Laplace实验室);University of California Riverside(加州大学河滨分校);University of Tennessee at Chattanooga(田纳西大学查塔努加分校)
Mitigating Illumination-Induced Domain Shift in Night-Time Pedestrian Detection for Intelligent Vehicles using Annotation-Preserving Diffusion Augmentation
DiffFAS: Face Anti-Spoofing via Generative Diffusion Models
Tianjin University(天津大学);Lappeenranta-Lahti University of Technology LUT(拉彭兰塔-拉赫蒂理工大学LUT);Great Bay University(大湾区大学);Xi’an Jiaotong University(西安交通大学)
Towards reconstructing experimental sparse-view X-ray CT data with diffusion models
1 Martin-Luther-University Halle-Wittenberg, Institute of Physics, Halle, Germany;2 Centrum Wiskunde \& Informatica, Computational Imaging Group, Amsterdam, The Netherlands
Adaptive Diffusion Priors for Accelerated MRI Reconstruction
Bilkent University(比尔肯特大学);National Magnetic Resonance Research Center (UMRAM)(国家磁共振研究中心(UMRAM));ASELSAN Research Center(ASELSAN研究中心);Amasya University(阿马西亚大学)
Regressor-Guided Image Editing Shifts Emotion and Disengagement Timing in Social Media
Eastern Switzerland University of Applied Sciences \& ETH Z\"urich St.Gallen Switzerland;ETH Z\"urich Z\"urich Switzerland;The Hong Kong University of Science;Eastern Switzerland University of Applied Sciences \& ETH Z\"urich;ETH Z\"urich
One-Step Residual Shifting Diffusion for Image Super-Resolution via Distillation
Kandinsky Lab(坎迪斯基实验室);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Luzin Research Center(卢津研究所);Moscow Independent Research Institute of Artificial Intelligence(莫斯科独立人工智能研究 institute);Applied AI Institute(应用人工智能研究所)