In recognition-based action interaction, robots' responses to human actions are often pre-designed according to recognized categories and thus stiff. In this paper, we specify a new Interactive Action Translation (IAT) task which aims to learn end-to-end action interaction from unlabeled interactive pairs, removing explicit action recognition. To enable learning on small-scale data, we propose a Paired-Embedding (PE) method for effective and reliable data augmentation. Specifically, our method first utilizes paired relationships to cluster individual actions in an embedding space. Then two actions originally paired can be replaced with other actions in their respective neighborhood, assembling into new pairs. An Act2Act network based on conditional GAN follows to learn from augmented data. Besides, IAT-test and IAT-train scores are specifically proposed for evaluating methods on our task. Experimental results on two datasets show impressive effects and broad application prospects of our method.
Kyushu University（九州大学）；Tianjin University of Technology（天津理工大学）；Alibaba Group（阿里巴巴集团）；Nanyang Technological University（南洋理工大学）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））
Online updating of the object model via samples from historical frames is of great importance for accurate visual object tracking. Recent works mainly focus on constructing effective and efficient updating methods while neglecting the training samples for learning discriminative object models, which is also a key part of a learning problem. In this paper, we propose the DeepMix that takes historical samples' embeddings as input and generates augmented embeddings online, enhancing the state-of-the-art online learning methods for visual object tracking. More specifically, we first propose the on…
Nanyang Technological University（南洋理工大学）；Singapore Management University（新加坡管理大学）；Salesforce Research Asia（赛富时亚洲研究院）
This paper investigates an open research problem of generating text-image pairs to improve the training of fine-grained image-to-text cross-modal retrieval task, and proposes a novel framework for paired data augmentation by uncovering the hidden semantic information of StyleGAN2 model. Specifically, we first train a StyleGAN2 model on the given dataset. We then project the real images back to the latent space of StyleGAN2 to obtain the latent codes. To make the generated images manipulatable, we further introduce a latent space alignment module to learn the alignment between StyleGAN2 latent …
Self-supervised learning has become a popular approach in recent years for its ability to learn meaningful representations without the need for data annotation. This paper proposes a novel image augmentation technique, overlaying images, which has not been widely applied in self-supervised learning. This method is designed to provide better guidance for the model to understand underlying information, resulting in more useful representations. The proposed method is evaluated using contrastive learning, a widely used self-supervised learning method that has shown solid performance in downstream …
This article presents an efficient way to produce feature-rich, high-dimensionality embedding spaces from real-life images. The features produced are designed to be independent from augmentations used in real-life cases which appear on social media. Our approach uses convolutional neural networks (CNN) to produce an embedding space. An ArcFace head was used to train the model by employing automatically produced augmentations. Additionally, we present a way to make an ensemble out of different embeddings containing the same semantic information, a way to normalize the resulting embedding using …
University College London（伦敦大学学院）；Brigham and Women's Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Broad Institute of Harvard and MIT（哈佛-麻省理工博德研究所）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）；University of Warwick（华威大学）；University Hospitals Coventry and Warwickshire NHS Trust（考文垂与沃里克郡国民保健服务信托大学医院）；The Alan Turing Institute（艾伦·图灵研究所）
Multiple Instance Learning (MIL) is a widely employed framework for learning on gigapixel whole-slide images (WSIs) from WSI-level annotations. In most MIL based analytical pipelines for WSI-level analysis, the WSIs are often divided into patches and deep features for patches (i.e., patch embeddings) are extracted prior to training to reduce the overall computational cost and cope with the GPUs' limited RAM. To overcome this limitation, we present EmbAugmenter, a data augmentation generative adversarial network (DA-GAN) that can synthesize data augmentations in the embedding space rather than …
University of Southern California（南加利福尼亚大学）；Clarifai Inc（克拉里菲人工智能公司）；Northeastern University（东北大学）
Recent advances in the field of artificial intelligence have been made possible by deep neural networks. In applications where data are scarce, transfer learning and data augmentation techniques are commonly used to improve the generalization of deep learning models. However, fine-tuning a transfer model with data augmentation in the raw input space has a high computational cost to run the full network for every augmented input. This is particularly critical when large models are implemented on embedded devices with limited computational and energy resources. In this work, we propose a method …
Learning the distance metric between pairs of samples has been studied for image retrieval and clustering. With the remarkable success of pair-based metric learning losses, recent works have proposed the use of generated synthetic points on metric learning losses for augmentation and generalization. However, these methods require additional generative networks along with the main network, which can lead to a larger model size, slower training speed, and harder optimization. Meanwhile, post-processing techniques, such as query expansion and database augmentation, have proposed the combination o…
Samsung AI Center Moscow（三星莫斯科人工智能中心）；Swiss Federal Institute of Technology Lausanne (EPFL)（洛桑瑞士联邦理工学院（EPFL））；National Research University Higher School of Economics（国立研究型大学高等经济学院）
Averaging predictions over a set of models -- an ensemble -- is widely used to improve predictive performance and uncertainty estimation of deep learning models. At the same time, many machine learning systems, such as search, matching, and recommendation systems, heavily rely on embeddings. Unfortunately, due to misalignment of features of independently trained models, embeddings, cannot be improved with a naive deep ensemble like approach. In this work, we look at the ensembling of representations and propose mean embeddings with test-time augmentation (MeTTA) simple yet well-performing reci…
Image text retrieval is a task to search for the proper textual descriptions of the visual world and vice versa. One challenge of this task is the vulnerability to input image and text corruptions. Such corruptions are often unobserved during the training, and degrade the retrieval model decision quality substantially. In this paper, we propose a novel image text retrieval technique, referred to as robust visual semantic embedding (RVSE), which consists of novel image-based and text-based augmentation techniques called semantic preserving augmentation for image (SPAugI) and text (SPAugT). Sinc…
Jilin University（吉林大学）；Megvii Inc.（旷视科技）；Beihang University（北京航空航天大学）；Huazhong University of Science and Technology（华中科技大学）；Tsinghua University（清华大学）
This paper considers learning deep features from long-tailed data. We observe that in the deep feature space, the head classes and the tail classes present different distribution patterns. The head classes have a relatively large spatial span, while the tail classes have significantly small spatial span, due to the lack of intra-class diversity. This uneven distribution between head and tail classes distorts the overall feature space, which compromises the discriminative ability of the learned features. Intuitively, we seek to expand the distribution of the tail classes by transferring from th…
Dali University（大理大学）；Liuzhou Vocational & Technical College（柳州职业技术学院）；The Second Xiangya Hospital of Central South University（中南大学湘雅二医院）；Hunan Key Laboratory of Early Diagnosis and Precise Treatment of Lung Cancer（湖南省肺癌早期诊断与精准治疗重点实验室）；Early-Stage Lung Cancer Center（早期肺癌中心）
Detection of pulmonary nodules by CT is used for screening lung cancer in early stages.omputer aided diagnosis (CAD) based on deep-learning method can identify the suspected areas of pulmonary nodules in CT images, thus improving the accuracy and efficiency of CT diagnosis. The accuracy and robustness of deep learning models. Method:In this paper, we explore (1) the data augmentation method based on the generation model and (2) the model structure improvement method based on the embedding mechanism. Two strategies have been introduced in this study: a new data augmentation method and a embeddi…
Improving Factuality of 3D Brain MRI Report Generation with Paired Image-domain Retrieval and Text-domain Augmentation
Cancer Biology, Seoul National University College of Medicine, Korea（首尔国立大学医学院癌症生物学系，韩国）；Radiology, Massachusetts General Hospital（麻省总医院放射科）；Harvard Medical School（哈佛医学院）；Biomedical Systems Informatics, Yonsei University College of Medicine, Korea（延世大学医学院生物医学系统信息学系，韩国）；Graduate School, Yonsei University, Korea（延世大学研究生院，韩国）；Radiology, Seoul National University College of Medicine, Korea（首尔国立大学医学院放射科，韩国）；Radiology, Seoul National University Hospital, Korea（首尔国立大学医院放射科，韩国）；Radiology, Seoul National University Bundang Hospital, Korea（首尔国立大学 Bundang 医院放射科，韩国）；Radiology, SMG-SNU Boramae Medical Center, Korea（SMG-SNU Boramae 医疗中心放射科，韩国）；Psychiatry, Yonsei University College of Medicine, Korea（延世大学医学院精神病学系，韩国）；Behavioral Sciences in Medicine, Yonsei University College of Medicine, Korea（延世大学医学院医学行为科学系，韩国）；Yonsei Institute for Digital Health, Korea（延世大学数字健康研究院，韩国）；Kim Jaechul Graduate School of AI, KAIST, Korea（金 Jaechul人工智能研究生院，韩国）
PairAug: What Can Augmented Image-Text Pairs Do for Radiology?
Australian Institute for Machine Learning (AIML)（澳大利亚机器学习研究所）；The University of Adelaide（阿德莱德大学）；South Australia Medical Imaging（南澳大利亚医学影像中心）；School of Computer Science and Engineering, Northwestern Polytechnical University（西北工业大学计算机科学与技术学院）；Ningbo Institute of Northwestern Polytechnical University（西北工业大学宁波研究院）
Many efforts have been devoted to designing sampling, mining, and weighting strategies in high-level deep metric learning (DML) loss objectives. However, little attention has been paid to low-level but essential data transformation. In this paper, we develop a novel mechanism, the independent domain embedding augmentation learning ({IDEAL}) method. It can simultaneously learn multiple independent embedding spaces for multiple domains generated by predefined data transformations. Our IDEAL is orthogonal to existing DML techniques and can be seamlessly combined with prior DML approaches for enha…
We present a method to learn a joint multimodal representation space that enables recognition of unseen activities in videos. We first compare the effect of placing various constraints on the embedding space using paired text and video data. We also propose a method to improve the joint embedding space using an adversarial formulation, allowing it to benefit from unpaired text and video data. By using unpaired text data, we show the ability to learn a representation that better captures unseen activities. In addition to testing on publicly available datasets, we introduce a new, large-scale te…
We introduce style augmentation, a new form of data augmentation based on random style transfer, for improving the robustness of convolutional neural networks (CNN) over both classification and regression based tasks. During training, our style augmentation randomizes texture, contrast and color, while preserving shape and semantic content. This is accomplished by adapting an arbitrary style transfer network to perform style randomization, by sampling input style embeddings from a multivariate normal distribution instead of inferring them from a style image. In addition to standard classificat…
Image Captioning, or the automatic generation of descriptions for images, is one of the core problems in Computer Vision and has seen considerable progress using Deep Learning Techniques. We propose to use Inception-ResNet Convolutional Neural Network as encoder to extract features from images, Hierarchical Context based Word Embeddings for word representations and a Deep Stacked Long Short Term Memory network as decoder, in addition to using Image Data Augmentation to avoid over-fitting. For data Augmentation, we use Horizontal and Vertical Flipping in addition to Perspective Transformations …
Google Research（谷歌研究院）；Mila（米拉研究所）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Brown University（布朗大学）
Pre-trained deep image representations are useful for post-training tasks such as classification through transfer learning, image retrieval, and object detection. Data augmentations are a crucial aspect of pre-training robust representations in both supervised and self-supervised settings. Data augmentations explicitly or implicitly promote invariance in the embedding space to the input image transformations. This invariance reduces generalization to those downstream tasks which rely on sensitivity to these particular data augmentations. In this paper, we propose a method of learning represent…
Self-supervised learning aims to learn a embedding space where semantically similar samples are close. Contrastive learning methods pull views of samples together and push different samples away, which utilizes semantic invariance of augmentation but ignores the relationship between samples. To better exploit the power of augmentation, we observe that semantically similar samples are more likely to have similar augmented views. Therefore, we can take the augmented views as a special description of a sample. In this paper, we model such a description as the augmentation distribution and we call…
Beijing Institute of Technology（北京理工大学）；Megvii Technology（旷视科技）；Beijing Normal University（北京师范大学）
Low-light raw denoising is an important and valuable task in computational photography where learning-based methods trained with paired real data are mainstream. However, the limited data volume and complicated noise distribution have constituted a learnability bottleneck for paired real data, which limits the denoising performance of learning-based methods. To address this issue, we present a learnability enhancement strategy to reform paired real data according to noise modeling. Our strategy consists of two efficient techniques: shot noise augmentation (SNA) and dark shading correction (DSC…
Lomonosov Moscow State University（罗蒙诺索夫莫斯科国立大学）；NRU Higher School of Economics（国立研究型大学高等经济学院）
Traffic sign recognition is a well-researched problem in computer vision. However, the state of the art methods works only for frequent sign classes, which are well represented in training datasets. We consider the task of rare traffic sign detection and classification. We aim to solve that problem by using synthetic training data. Such training data is obtained by embedding synthetic images of signs in the real photos. We propose three methods for making synthetic signs consistent with a scene in appearance. These methods are based on modern generative adversarial network (GAN) architectures.…
Centre for Secure Information Technologies (CSIT)（安全信息技术中心（CSIT））；Queen’s University Belfast（贝尔法斯特女王大学）
In this paper we study data augmentation for opcode sequence based Android malware detection. Data augmentation has been successfully used in many areas of deep-learning to significantly improve model performance. Typically, data augmentation simulates realistic variations in data to increase the apparent diversity of the training-set. However, for opcode-based malware analysis it is not immediately clear how to apply data augmentation. Hence we first study the use of fixed transformations, then progress to adaptive methods. We propose a novel data augmentation method -- Self-Embedding Languag…
Re-mixing Embeddings for Patient Augmentation in Data Scarce Multiple Instance Learning
Computational Health Center & Helmholtz AI, Helmholtz Munich（计算健康中心与亥姆霍兹AI，亥姆霍兹慕尼黑）；Computer Aided Medical Procedures, Technical University of Munich（计算机辅助医疗程序，慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Medicine III, Ludwig Maximilian University Hospital（路德维希·马克西米利安大学医院第三内科）；Department of Physics, Ludwig-Maximilians-Universität München（慕尼黑路德维希-马克西米利安大学物理系）；DKTK, German Cancer Consortium, partner site Munich（德国癌症联盟DKTK，慕尼黑合作站点）
CLIP-PAE: Projection-Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable, and Controllable Text-Guided Face Manipulation
In the autonomous driving domain, data collection and annotation from real vehicles are expensive and sometimes unsafe. Simulators are often used for data augmentation, which requires realistic sensor models that are hard to formulate and model in closed forms. Instead, sensors models can be learned from real data. The main challenge is the absence of paired data set, which makes traditional supervised learning techniques not suitable. In this work, we formulate the problem as image translation from unpaired data and employ CycleGANs to solve the sensor modeling problem for LiDAR, to produce r…
Convolutional neural networks (CNN) for image steganalysis demonstrate better performances with employing concepts from high-level vision tasks. The major employed concept is to use data augmentation to avoid overfitting due to limited data. To augment data without damaging the message embedding, only rotating multiples of 90 degrees or horizontally flipping are used in steganalysis, which generates eight fixed results from one sample. To overcome this limitation, we propose BitMix, a data augmentation method for spatial image steganalysis. BitMix mixes a cover and stego image pair by swapping…
Valeo Egypt AI Research（法雷奥埃及人工智能研究院）；Cairo University（开罗大学）
Data scarcity is a bottleneck to machine learning-based perception modules, usually tackled by augmenting real data with synthetic data from simulators. Realistic models of the vehicle perception sensors are hard to formulate in closed form, and at the same time, they require the existence of paired data to be learned. In this work, we propose two unsupervised neural sensor models based on unpaired domain translations with CycleGANs and Neural Style Transfer techniques. We employ CARLA as the simulation environment to obtain simulated LiDAR point clouds, together with their annotations for dat…
University of Science and Technology of China（中国科学技术大学）；Tencent Corporation（腾讯公司）；Georgia Institute of Technology（佐治亚理工学院）；The Chinese University of Hong Kong（香港中文大学）；University of Enna Kore（恩纳科雷大学）
In this paper, we propose two techniques, namely joint modeling and data augmentation, to improve system performances for audio-visual scene classification (AVSC). We employ pre-trained networks trained only on image data sets to extract video embedding; whereas for audio embedding models, we decide to train them from scratch. We explore different neural network architectures for joint modeling to effectively combine the video and audio modalities. Moreover, data augmentation strategies are investigated to increase audio-visual training set size. For the video modality the effectiveness of sev…
