Image generation abilities of text-to-image diffusion models have significantly advanced, yielding highly photo-realistic images from descriptive text and increasing the viability of leveraging synthetic images to train computer vision models. To serve as effective training data, generated images must be highly realistic while also sufficiently diverse within the support of the target data distribution. Yet, state-of-the-art conditional image generation models have been primarily optimized for creative applications, prioritizing image realism and prompt adherence over conditional diversity. In this paper, we investigate how to improve the diversity of generated images with the goal of increasing their effectiveness to train downstream image classification models, without fine-tuning the image generation model. We find that conditioning the generation process on an augmented real image and text prompt produces generations that serve as effective synthetic datasets for downstream training. Conditioning on real training images contextualizes the generation process to produce images that are in-domain with the real image distribution, while data augmentations introduce visual diversity that improves the performance of the downstream classifier. We validate augmentation-conditioning on a total of five established long-tail and few-shot image classification benchmarks and show that leveraging augmentations to condition the generation process results in consistent improvements over the state-of-the-art on the long-tailed benchmark and remarkable gains in extreme few-shot regimes of the remaining four benchmarks. These results constitute an important step towards effectively leveraging synthetic data for downstream training.
Generative Augmented Reality: Paradigms, Technologies, and Future Applications
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；The Hong Kong University of Science and Technology（香港科学与技术大学）；Nanyang Technological University（南洋理工大学）；XMax.AI Ltd.（XMax.AI有限公司）
Hand gestures are a natural means of interaction in Augmented Reality and Virtual Reality (AR/VR) applications. Recently, there has been an increased focus on removing the dependence of accurate hand gesture recognition on complex sensor setup found in expensive proprietary devices such as the Microsoft HoloLens, Daqri and Meta Glasses. Most such solutions either rely on multi-modal sensor data or deep neural networks that can benefit greatly from abundance of labelled data. Datasets are an integral part of any deep learning based research. They have been the principal reason for the substanti…
Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation
Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy（人工智能与计算机系统单位，工程系，罗马生物医学大学，意大利罗马）；Department of Diagnostics and Intervention, Biomedical Engineering and Radiation Physics, Umeå University, Umeå, Sweden（诊断与介入系，生物医学工程与放射物理，乌梅大学，瑞典乌梅）
GeoBridge: Decoupled Semantic Conditioning for Generative Image Geolocalization
University of Chinese Academy of Sciences（中国科学院大学）；School of Advanced Interdisciplinary Sciences（先进交叉科学学院）；School of Electronic, Electrical and Communication Engineering（电子电气与通信工程学院）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）
Image classification datasets are often imbalanced, characteristic that negatively affects the accuracy of deep-learning classifiers. In this work we propose balancing GAN (BAGAN) as an augmentation tool to restore balance in imbalanced datasets. This is challenging because the few minority-class images may not be enough to train a GAN. We overcome this issue by including during the adversarial training all available images of majority and minority classes. The generative model learns useful features from majority classes and uses these to generate images for minority classes. We apply class c…
University of Houston（休斯顿大学）；National Institutes of Health（美国国立卫生研究院）；Johns Hopkins University（约翰斯·霍普金斯大学）
Deep Neural Networks (or DNNs) must constantly cope with distribution changes in the input data when the task of interest or the data collection protocol changes. Retraining a network from scratch to combat this issue poses a significant cost. Meta-learning aims to deliver an adaptive model that is sensitive to these underlying distribution changes, but requires many tasks during the meta-training process. In this paper, we propose a tAsk-auGmented actIve meta-LEarning (AGILE) method to efficiently adapt DNNs to new tasks by using a small number of training examples. AGILE combines a meta-lear…
Mississippi State University（密西西比州立大学）；University of Missouri-Columbia（密苏里大学哥伦比亚分校）
A significant challenge in object detection is accurate identification of an object's position in image space, whereas one algorithm with one set of parameters is usually not enough, and the fusion of multiple algorithms and/or parameters can lead to more robust results. Herein, a new computational intelligence fusion approach based on the dynamic analysis of agreement among object detection outputs is proposed. Furthermore, we propose an online versus just in training image augmentation strategy. Experiments comparing the results both with and without fusion are presented. We demonstrate that…
A Generalized Unbiased Risk Estimator for Learning with Augmented Classes
Chongqing University（重庆大学）；University of Electronic Science and Technology of China（电子科技大学）；Zhejiang University（浙江大学）；Nanyang Technological University（南洋理工大学）
Semantic scene completion (SSC) is a challenging Computer Vision task with many practical applications, from robotics to assistive computing. Its goal is to infer the 3D geometry in a field of view of a scene and the semantic labels of voxels, including occluded regions. In this work, we present SPAwN, a novel lightweight multimodal 3D deep CNN that seamlessly fuses structural data from the depth component of RGB-D images with semantic priors from a bimodal 2D segmentation network. A crucial difficulty in this field is the lack of fully labeled real-world 3D datasets which are large enough to …
RASR: Retrieval-Augmented Super Resolution for Practical Reference-based Image Restoration
State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究院（TeleAI），中国电信）；State Key Laboratory of Internet of Things for Smart City（物联网智慧城市国家重点实验室）；Department of Computer Science and Information Science, University of Macau（澳门大学计算机科学与信息科学系）
The University of Hong Kong（香港大学）；Xiamen University（厦门大学）
Self-supervised representation learning has been extremely successful in medical image analysis, as it requires no human annotations to provide transferable representations for downstream tasks. Recent self-supervised learning methods are dominated by noise-contrastive estimation (NCE, also known as contrastive learning), which aims to learn invariant visual representations by contrasting one homogeneous image pair with a large number of heterogeneous image pairs in each training step. Nonetheless, NCE-based approaches still suffer from one major problem that is one homogeneous pair is not eno…
Institute of Information Science, Academia Sinica（中央研究院信息科学研究所）；Tencent America（腾讯美国）；Research Center for Information Technology Innovation, Academia Sinica（中央研究院信息科技创新研究中心）；Tainan National University of the Arts（台南艺术大学）
This paper aims at recognizing partially observed human actions in videos. Action videos acquired in uncontrolled environments often contain corrupt frames, which make actions partially observed. Furthermore, these frames can last for arbitrary lengths of time and appear irregularly. They are inconsistent with training data and degrade the performance of pre-trained action recognition systems. We present an approach to address this issue. For each training and testing actions, we divide it into segments and explore the mutual dependency between temporal segments. This property states that the …
