Image text retrieval is a task to search for the proper textual descriptions of the visual world and vice versa. One challenge of this task is the vulnerability to input image and text corruptions. Such corruptions are often unobserved during the training, and degrade the retrieval model decision quality substantially. In this paper, we propose a novel image text retrieval technique, referred to as robust visual semantic embedding (RVSE), which consists of novel image-based and text-based augmentation techniques called semantic preserving augmentation for image (SPAugI) and text (SPAugT). Since SPAugI and SPAugT change the original data in a way that its semantic information is preserved, we enforce the feature extractors to generate semantic aware embedding vectors regardless of the corruption, improving the model robustness significantly. From extensive experiments using benchmark datasets, we show that RVSE outperforms conventional retrieval schemes in terms of image-text retrieval performance.
Logo retrieval is a challenging problem since the definition of similarity is more subjective compared to image retrieval tasks and the set of known similarities is very scarce. To tackle this challenge, in this paper, we propose a simple but effective segment-based augmentation strategy to introduce artificially similar logos for training deep networks for logo retrieval. In this novel augmentation strategy, we first find segments in a logo and apply transformations such as rotation, scaling, and color change, on the segments, unlike the conventional image-level augmentation strategies. Moreo…
An important component of human analysis of medical images and their context is the ability to relate newly seen things to related instances in our memory. In this paper we mimic this ability by using multi-modal retrieval augmentation and apply it to several tasks in chest X-ray analysis. By retrieving similar images and/or radiology reports we expand and regularize the case at hand with additional knowledge, while maintaining factual knowledge consistency. The method consists of two components. First, vision and language modalities are aligned using a pre-trained CLIP model. To enforce that …
Off-the-shelf convolutional neural network features achieve outstanding results in many image retrieval tasks. However, their invariance to target data is pre-defined by the network architecture and training data. Existing image retrieval approaches require fine-tuning or modification of pre-trained networks to adapt to variations unique to the target data. In contrast, our method enhances the invariance of off-the-shelf features by aggregating features extracted from images augmented at test-time, with augmentations guided by a policy learned through reinforcement learning. The learned policy…
Nanyang Technological University（南洋理工大学）；Singapore Management University（新加坡管理大学）；Salesforce Research Asia（赛富时亚洲研究院）
This paper investigates an open research problem of generating text-image pairs to improve the training of fine-grained image-to-text cross-modal retrieval task, and proposes a novel framework for paired data augmentation by uncovering the hidden semantic information of StyleGAN2 model. Specifically, we first train a StyleGAN2 model on the given dataset. We then project the real images back to the latent space of StyleGAN2 to obtain the latent codes. To make the generated images manipulatable, we further introduce a latent space alignment module to learn the alignment between StyleGAN2 latent …
University of Texas Austin（德克萨斯大学奥斯汀分校）；Facebook Inc（脸书公司）
Data augmentation (DA) is an essential technique for training state-of-the-art deep learning systems. In this paper, we empirically show data augmentation might introduce noisy augmented examples and consequently hurt the performance on unaugmented data during inference. To alleviate this issue, we propose a simple yet highly effective approach, dubbed \emph{KeepAugment}, to increase augmented images fidelity. The idea is first to use the saliency map to detect important regions on the original images and then preserve these informative regions during augmentation. This information-preserving …
Thanks to the success of object detection technology, we can retrieve objects of the specified classes even from huge image collections. However, the current state-of-the-art object detectors (such as Faster R-CNN) can only handle pre-specified classes. In addition, large amounts of positive and negative visual samples are required for training. In this paper, we address the problem of open-vocabulary object retrieval and localization, where the target object is specified by a textual query (e.g., a word or phrase). We first propose Query-Adaptive R-CNN, a simple extension of Faster R-CNN adap…
Pose estimation is usually tackled as either a bin classification or a regression problem. In both cases, the idea is to directly predict the pose of an object. This is a non-trivial task due to appearance variations between similar poses and similarities between dissimilar poses. Instead, we follow the key idea that comparing two poses is easier than directly predicting one. Render-and-compare approaches have been employed to that end, however, they tend to be unstable, computationally expensive, and slow for real-time applications. We propose doing category-level pose estimation by learning …
Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）；University of Udine（乌迪内大学）；Free University of Bozen-Bolzano（博尔扎诺自由大学）
Every hour, huge amounts of visual contents are posted on social media and user-generated content platforms. To find relevant videos by means of a natural language query, text-video retrieval methods have received increased attention over the past few years. Data augmentation techniques were introduced to increase the performance on unseen test examples by creating new training samples with the application of semantics-preserving techniques, such as color space or geometric transformations on images. Yet, these techniques are usually applied on raw data, leading to more resource-demanding solu…
Phase retrieval is an important problem with significant physical and industrial applications. In this paper, we consider the case where the magnitude of the measurement of an underlying signal is corrupted by Gaussian noise. We introduce a convex augmentation approach for phase retrieval based on total variation regularization. In contrast to popular convex relaxation models like PhaseLift, our model can be efficiently solved by a modified semi-proximal alternating direction method of multipliers (sPADMM). The modified sPADMM is more general and flexible than the standard one, and its converg…
Shape deformation of targets in SAR image due to random orientation and partial information loss caused by occlusion of the radar signal, is an essential challenge in SAR ship detection. In this paper, we propose a data augmentation method to train a deep network that is robust to partial information loss within the targets. Taking advantage of ground-truth annotations for bounding box and instance segmentation mask, we present a simple and effective pipeline to simulate information loss on targets in instance-level, while preserving contextual information. Furthermore, we adopt deformable con…
CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities
NepAl Applied Mathematics and Informatics Institute for research（尼泊尔应用数学与信息学研究所）；Fogsphere (Redev.AI)（Fogsphere（Redev.AI））；University of Lausanne（洛桑大学）；West Virginia University（西弗吉尼亚大学）；University of Aberdeen（阿伯丁大学）
Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves substantial improvement in performance on image-caption retrieval w.r.t. similar methods. Second, we show that retrieval-augmented multi-modal transformers usin…
IPF-RDA: An Information-Preserving Framework for Robust Data Augmentation
State Key Laboratory for Novel Software Technology, Department of Computer Science and Technology, Nanjing University（新型软件技术国家重点实验室，计算机科学与技术系，南京大学）；State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）；School of Electronic Science and Engineering, Nanjing University（电子科学与工程学院，南京大学）
City University of Hong Kong（香港城市大学）；Tencent AI Lab（腾讯人工智能实验室）；Shenzhen University（深圳大学）
Deep hashing models have been proposed as an efficient method for large-scale similarity search. However, most existing deep hashing methods only utilize fine-level labels for training while ignoring the natural semantic hierarchy structure. This paper presents an effective method that preserves the classwise similarity of full-level semantic hierarchy for large-scale image retrieval. Experiments on two benchmark datasets show that our method helps improve the fine-level retrieval performance. Moreover, with the help of the semantic hierarchy, it can produce significantly better binary codes f…
Deep neural networks have achieved state-of-the-art results in various vision and/or language tasks. Despite the use of large training datasets, most models are trained by iterating over single input-output pairs, discarding the remaining examples for the current prediction. In this work, we actively exploit the training data, using the information from nearest training examples to aid the prediction both during training and testing. Specifically, our approach uses the target of the most similar training example to initialize the memory state of an LSTM model, or to guide attention mechanisms.…
MADTempo: An Interactive System for Multi-Event Temporal Video Retrieval with Query Augmentation
Hanoi University of Science and Technology（河内科技大学）；The University of Danang - University of Science and Technology（丹江-科技大学）；Vietnam National University（越南国家大学）
Image retrieval outperforms diffusion models on data augmentation
Amazon Web Services（亚马逊云科技）；Mirelo AI（Mirelo人工智能公司）；ELLIS Institute Tübingen（图宾根欧洲学习与智能系统研究所）；GlaxoSmithKline（葛兰素史克）；Amazon（亚马逊）；Institute of Science and Technology Austria（奥地利科学技术研究所）；Oxford Internet Institute, University of Oxford（牛津大学牛津互联网研究院）；International Max Planck Research School for Intelligent Systems（国际马克斯·普朗克智能系统研究学院）；Institute of Computer Science and Campus Institute Data Science, University of Göttingen（哥廷根大学计算机科学研究所与校园数据科学研究所）；Tübingen AI Center, University of Tübingen（图宾根大学图宾根人工智能中心）
University of Toronto（多伦多大学）；Vector Institute（向量学院）；Google Research（谷歌研究院）
We introduce Exemplar VAEs, a family of generative models that bridge the gap between parametric and non-parametric, exemplar based generative models. Exemplar VAE is a variant of VAE with a non-parametric prior in the latent space based on a Parzen window estimator. To sample from it, one first draws a random exemplar from a training set, then stochastically transforms that exemplar into a latent code and a new observation. We propose retrieval augmented training (RAT) as a way to speed up Exemplar VAE training by using approximate nearest neighbor search in the latent space to define a lower…
Megvii Technology（旷视科技）；Tsinghua University（清华大学）；University of Central Florida（中佛罗里达大学）
In this paper, we present new data pre-processing and augmentation techniques for DNN-based raw image denoising. Compared with traditional RGB image denoising, performing this task on direct camera sensor readings presents new challenges such as how to effectively handle various Bayer patterns from different data sources, and subsequently how to perform valid data augmentation with raw images. To address the first problem, we propose a Bayer pattern unification (BayerUnify) method to unify different Bayer patterns. This allows us to fully utilize a heterogeneous dataset to train a single denoi…
INESC-ID（葡萄牙国家系统与计算机工程研究所）；Instituto Superior Técnico（高等技术学院）；University of Lisbon（里斯本大学）；University of Copenhagen（哥本哈根大学）；Pioneer Center for AI（先锋人工智能中心）
Recent advances in image captioning have focused on scaling the data and model size, substantially increasing the cost of pre-training and finetuning. As an alternative to large models, we present SmallCap, which generates a caption conditioned on an input image and related captions retrieved from a datastore. Our model is lightweight and fast to train, as the only learned parameters are in newly introduced cross-attention layers between a pre-trained CLIP encoder and GPT-2 decoder. SmallCap can transfer to new domains without additional finetuning and can exploit large-scale data in a trainin…
SGD-Mix: Enhancing Domain-Specific Image Classification with Label-Preserving Data Augmentation
Department of Computer Science and Information Engineering, National Cheng Kung University（计算机科学与信息工程系，国立成功大学）；Department of Biomedical Informatics, Harvard Medical School, Harvard University（生物医学信息学系，哈佛医学院，哈佛大学）
In this paper, we propose a privacy-preserving image-retrieval scheme using a codebook generated by using a plain-image dataset. Encryption-then-compression (EtC) images, which were proposed for EtC systems, have been used in conventional privacy-preserving image-retrieval schemes, in which a codebook is generated from EtC images uploaded by image owners, and extended SIMPLE descriptors are then calculated as image descriptors by using the codebook. In contrast, in the proposed scheme, a codebook is generated from a dataset independent of uploaded images. The use of an independent dataset enab…
The Efficacy of Semantics-Preserving Transformations in Self-Supervised Learning for Medical Ultrasound
Cheriton School of Computer Science, University of Waterloo（滑铁卢大学计算机科学学院）；Department of Systems Design Engineering, University of Waterloo（滑铁卢大学系统设计工程系）；Schulich School of Medicine and Dentistry, Western University（西部大学施乐希医学与牙科学校）
Egocentric visual context detection can support intelligence augmentation applications. We created a wearable system, called PAL, for wearable, personalized, and privacy-preserving egocentric visual context detection. PAL has a wearable device with a camera, heart-rate sensor, on-device deep learning, and audio input/output. PAL also has a mobile/web application for personalized context labeling. We used on-device deep learning models for generic object and face detection, low-shot custom face and context recognition (e.g., activities like brushing teeth), and custom context clustering (e.g., …
