Generative Adversarial Networks (GANs) are machine learning methods that are used in many important and novel applications. For example, in imaging science, GANs are effectively utilized in generating image datasets, photographs of human faces, image and video captioning, image-to-image translation, text-to-image translation, video prediction, and 3D object generation to name a few. In this paper, we discuss how GANs can be used to create an artificial world. More specifically, we discuss how GANs help to describe an image utilizing image/video captioning methods and how to translate the image to a new image using image-to-image translation frameworks in a theme we desire. We articulate how GANs impact creating a customized world.
Memorial University of Newfoundland（纽芬兰纪念大学）；University of Waterloo（滑铁卢大学）
In this paper we present a method for algorithmic melody generation using a generative adversarial network without recurrent components. Music generation has been successfully done using recurrent neural networks, where the model learns sequence information that can help create authentic sounding melodies. Here, we use DC-GAN architecture with dilated convolutions and towers to capture sequential information as spatial image information, and learn long-range dependencies in fixed-length melody forms such as Irish traditional reel.
Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）；Google Research（谷歌研究院）；University of KwaZulu-Natal（夸祖鲁-纳塔尔大学）
Inferring model parameters from experimental data is a grand challenge in many sciences, including cosmology. This often relies critically on high fidelity numerical simulations, which are prohibitively computationally expensive. The application of deep learning techniques to generative modeling is renewing interest in using high dimensional density estimators as computationally inexpensive emulators of fully-fledged simulations. These generative models have the potential to make a dramatic shift in the field of scientific simulations, but for that shift to happen we need to study the performa…
National Engineering Lab for Video Technology（视频技术国家工程实验室）；Peking University（北京大学）；The Future Lab, Tsinghua University（清华大学未来实验室）
We present a method for improving human design of chairs. The goal of the method is generating enormous chair candidates in order to facilitate human designer by creating sketches and 3d models accordingly based on the generated chair design. It consists of an image synthesis module, which learns the underlying distribution of training dataset, a super-resolution module, which improve quality of generated image and human involvements. Finally, we manually pick one of the generated candidates to create a real life chair for illustration.
American University of Beirut（贝鲁特美国大学）；Reinstein Vision；London Vision Clinic（伦敦视力诊所）；Columbia University Medical Center（哥伦比亚大学医学中心）；Sorbonne Université（索邦大学）；Biomedical Science Research Institute, Ulster University（阿尔斯特大学生物医学科学研究所）；Bascom Palmer Eye Institute, University of Miami Miller School of Medicine（迈阿密大学米勒医学院巴斯康帕尔默眼科研究所）；Stanford University（斯坦福大学）；Saint Joseph University（圣约瑟夫大学）；American University of Beirut Medical Center（贝鲁特美国大学医学中心）
Universidad Politécnica de Madrid（马德里理工大学）；Universidad de Jaén（哈恩大学）；Universidad de Ciego de Ávila（谢戈德阿维拉大学）
Research and education in machine learning needs diverse, representative, and open datasets that contain sufficient samples to handle the necessary training, validation, and testing tasks. Currently, the Recommender Systems area includes a large number of subfields in which accuracy and beyond accuracy quality measures are continuously improved. To feed this research variety, it is necessary and convenient to reinforce the existing datasets with synthetic ones. This paper proposes a Generative Adversarial Network (GAN)-based method to generate collaborative filtering datasets in a parameterize…
Image translation across domains for unpaired datasets has gained interest and great improvement lately. In medical imaging, there are multiple imaging modalities, with very different characteristics. Our goal is to use cross-modality adaptation between CT and MRI whole cardiac scans for semantic segmentation. We present a segmentation network using synthesised cardiac volumes for extremely limited datasets. Our solution is based on a 3D cross-modality generative adversarial network to share information between modalities and generate synthesized data using unpaired datasets. Our network utili…
Universitat de Barcelona（巴塞罗那大学）；Erasmus MC（伊拉斯姆斯医学中心）；Imperial College London（帝国理工学院）
Despite technological and medical advances, the detection, interpretation, and treatment of cancer based on imaging data continue to pose significant challenges. These include inter-observer variability, class imbalance, dataset shifts, inter- and intra-tumour heterogeneity, malignancy determination, and treatment effect uncertainty. Given the recent advancements in Generative Adversarial Networks (GANs), data synthesis, and adversarial training, we assess the potential of these technologies to address a number of key challenges of cancer imaging. We categorise these challenges into (a) data s…
Nanjing University of Information Science and Technology（南京信息工程大学）；Radboud University Medical Center（拉德堡德大学医学中心）；Shandong University (Weihai)（山东大学（威海））
Missing scans are inevitable in longitudinal studies due to either subject dropouts or failed scans. In this paper, we propose a deep learning framework to predict missing scans from acquired scans, catering to longitudinal infant studies. Prediction of infant brain MRI is challenging owing to the rapid contrast and structural changes particularly during the first year of life. We introduce a trustworthy metamorphic generative adversarial network (MGAN) for translating infant brain MRI from one time-point to another. MGAN has three key features: (i) Image translation leveraging spatial and fre…
Creating Power Distribution Network Layouts Using Generative Adversarial Networks and Image-Based Representations
School of Engineering (ICAI), Comillas Pontifical University（工程学院（ICAI），卡利斯大学）；Institute for Research in Technology (IIT), School of Engineering (ICAI), Comillas Pontifical University（技术研究所（IIT），工程学院（ICAI），卡利斯大学）
Generative adversarial networks have been able to generate striking results in various domains. This generation capability can be general while the networks gain deep understanding regarding the data distribution. In many domains, this data distribution consists of anomalies and normal data, with the anomalies commonly occurring relatively less, creating datasets that are imbalanced. The capabilities that generative adversarial networks offer can be leveraged to examine these anomalies and help alleviate the challenge that imbalanced datasets propose via creating synthetic anomalies. This anom…
Columbia University（哥伦比亚大学）；Stanford University（斯坦福大学）
Generative adversarial networks, which can generate metasurfaces based on a training set of high performance device layouts, have the potential to significantly reduce the computational cost of the metasurface design process. However, basic GAN architectures are unable to fully capture the detailed features of topologically complex metasurfaces, and generated devices therefore require additional computationally-expensive design refinement. In this Letter, we show that GANs can better learn spatially fine features from high-resolution training data by progressively growing its network architect…
Université Côte d’Azur（蔚蓝海岸大学）；Inria Sophia Antipolis Méditerranée（法国国家信息与自动化研究所索菲亚·安蒂波利斯-地中海研究中心）；Head and Neck University Institute, Nice University Hospital（尼斯大学医院头颈大学研究所）；Department of Radiology, Nice University Hospital（尼斯大学医院放射科）；Oticon Medical（奥迪康医疗公司）
Metal Artifacts creates often difficulties for a high quality visual assessment of post-operative imaging in {c}omputed {t}omography (CT). A vast body of methods have been proposed to tackle this issue, but {these} methods were designed for regular CT scans and their performance is usually insufficient when imaging tiny implants. In the context of post-operative high-resolution {CT} imaging, we propose a 3D metal {artifact} reduction algorithm based on a generative adversarial neural network. It is based on the simulation of physically realistic CT metal artifacts created by cochlea implant el…
Lesion synthesis received much attention with the rise of efficient generative models for augmenting training data, drawing lesion evolution scenarios, or aiding expert training. The quality and diversity of synthesized data are highly dependent on the annotated data used to train the models, which not rarely struggle to derive very different yet realistic samples from the training ones. That adds an inherent bias to lesion segmentation algorithms and limits synthesizing lesion evolution scenarios efficiently. This paper presents a method for decoupling shape and density for liver lesion synth…
The impressive success of Generative Adversarial Networks (GANs) is often overshadowed by the difficulties in their training. Despite the continuous efforts and improvements, there are still open issues regarding their convergence properties. In this paper, we propose a simple training variation where suitable weights are defined and assist the training of the Generator. We provide theoretical arguments why the proposed algorithm is better than the baseline training in the sense of speeding up the training process and of creating a stronger Generator. Performance results showed that the new al…
The huge amount of video data produced daily by camera-based systems, such as surveilance, medical and telecommunication systems, emerges the need for effective video summarization (VS) methods. These methods should be capable of creating an overview of the video content. In this paper, we propose a novel VS method based on a Generative Adversarial Network (GAN) model pre-trained with human eye fixations. The main contribution of the proposed method is that it can provide perceptually compatible video summaries by combining both perceived color and spatiotemporal visual attention cues in a uns…
"Art is the lie that enables us to realize the truth." - Pablo Picasso. For centuries, humans have dedicated themselves to producing arts to convey their imagination. The advancement in technology and deep learning in particular, has caught the attention of many researchers trying to investigate whether art generation is possible by computers and algorithms. Using generative adversarial networks (GANs), applications such as synthesizing photorealistic human faces and creating captions automatically from images were realized. This survey takes a comprehensive look at the recent works using GANs…
Generative adversarial networks (GANs) have many application areas including image editing, domain translation, missing data imputation, and support for creative work. However, GANs are considered 'black boxes'. Specifically, the end-users have little control over how to improve editing directions through disentanglement. Prior work focused on new GAN architectures to disentangle editing directions. Alternatively, we propose GANravel a user-driven direction disentanglement tool that complements the existing GAN architectures and allows users to improve editing directions iteratively. In two us…
University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；Massachusetts Institute of Technology（麻省理工学院）；University of Central Florida（中佛罗里达大学）
Metasurfaces have enabled precise electromagnetic wave manipulation with strong potential to obtain unprecedented functionalities and multifunctional behavior in flat optical devices. These advantages in precision and functionality come at the cost of tremendous difficulty in finding individual meta-atom structures based on specific requirements (commonly formulated in terms of electromagnetic responses), which makes the design of multifunctional metasurfaces a key challenge in this field. In this paper, we present a Generative Adversarial Networks (GAN) that can tackle this problem and genera…
Non-intrusive load monitoring (NILM) allows users and energy providers to gain insight into home appliance electricity consumption using only the building's smart meter. Most current techniques for NILM are trained using significant amounts of labeled appliances power data. The collection of such data is challenging, making data a major bottleneck in creating well generalizing NILM solutions. To help mitigate the data limitations, we present the first truly synthetic appliance power signature generator. Our solution, PowerGAN, is based on conditional, progressively growing, 1-D Wasserstein gen…
Meteorology satellite visible light images is critical for meteorology support and forecast. However, there is no such kind of data during night time. To overcome this, we propose a method based on deep learning to create synthetic satellite visible light images during night. Specifically, to produce more realistic products, we train a Generative Adversarial Networks (GAN) model to generate visible light images given the corresponding satellite infrared images and numerical weather prediction(NWP) products. To better model the nonlinear relationship from infrared data and NWP products to visib…
Since their proposal in the 2014 paper by Ian Goodfellow, there has been an explosion of research into the area of Generative Adversarial Networks. While they have been utilised in many fields, the realm of malware research is a problem space in which GANs have taken root. From balancing datasets to creating unseen examples in rare classes, GAN models offer extensive opportunities for application. This paper surveys the current research and literature for the use of Generative Adversarial Networks in the malware problem space. This is done with the hope that the reader may be able to gain an o…
Radboud University（拉德堡德大学）；DPG Media Online Services（DPG媒体在线服务公司）
The goal of this work is to help mitigate the already existing gender wage gap by supplying unbiased job recommendations based on resumes from job seekers. We employ a generative adversarial network to remove gender bias from word2vec representations of 12M job vacancy texts and 900k resumes. Our results show that representations created from recruitment texts contain algorithmic bias and that this bias results in real-world consequences for recommendation systems. Without controlling for bias, women are recommended jobs with significantly lower salary in our data. With adversarially fair repr…
Synthesising photo-realistic images from natural language is one of the challenging problems in computer vision. Over the past decade, a number of approaches have been proposed, of which the improved Stacked Generative Adversarial Network (StackGAN-v2) has proven capable of generating high resolution images that reflect the details specified in the input text descriptions. In this paper, we aim to assess the robustness and fault-tolerance capability of the StackGAN-v2 model by introducing variations in the training data. However, due to the working principle of Generative Adversarial Network (…
Stanford University（斯坦福大学）；Google AI Applied Science（谷歌AI应用科学部）
A key challenge in metasurface design is the development of algorithms that can effectively and efficiently produce high performance devices. Design methods based on iterative optimization can push the performance limits of metasurfaces, but they require extensive computational resources that limit their implementation to small numbers of microscale devices. We show that generative neural networks can train from images of periodic, topology-optimized metagratings to produce high-efficiency, topologically complex devices operating over a broad range of deflection angles and wavelengths. Further…
A common problem when forecasting rare events, such as recessions, is limited data availability. Recent advancements in deep learning and generative adversarial networks (GANs) make it possible to produce high-fidelity synthetic data in large quantities. This paper uses a model called DoppelGANger, a GAN tailored to producing synthetic time series data, to generate synthetic Treasury yield time series and associated recession indicators. It is then shown that short-range forecasting performance for Treasury yields is improved for models trained on synthetic data relative to models trained only…
A common approach to generating symbolic music using neural networks involves repeated sampling of an autoregressive model until the full output sequence is obtained. While such approaches have shown some promise in generating short sequences of music, this typically has not extended to cases where the final target sequence is significantly longer, for example an entire piece of music. In this work we propose a network trained in an adversarial process to generate entire pieces of solo shakuhachi music, in the form of symbolic notation. The pieces are intended to refer clearly to traditional s…
Neuromorphic data carries information in spatio-temporal patterns encoded by spikes. Accordingly, a central problem in neuromorphic computing is training spiking neural networks (SNNs) to reproduce spatio-temporal spiking patterns in response to given spiking stimuli. Most existing approaches model the input-output behavior of an SNN in a deterministic fashion by assigning each input to a specific desired output spiking sequence. In contrast, in order to fully leverage the time-encoding capacity of spikes, this work proposes to train SNNs so as to match distributions of spiking signals rather …
Carnegie Mellon University（卡内基梅隆大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Much research effort has been devoted to better understanding adversarial examples, which are specially crafted inputs to machine-learning models that are perceptually similar to benign inputs, but are classified differently (i.e., misclassified). Both algorithms that create adversarial examples and strategies for defending against them typically use $L_p$-norms to measure the perceptual similarity between an adversarial input and its benign original. Prior work has already shown, however, that two images need not be close to each other as measured by an $L_p$-norm to be perceptually similar. …
Generating realistic images is difficult, and many formulations for this task have been proposed recently. If we restrict the task to that of generating a particular class of images, however, the task becomes more tractable. That is to say, instead of generating an arbitrary image as a sample from the manifold of natural images, we propose to sample images from a particular "subspace" of natural images, directed by a low-resolution image from the same subspace. The problem we address, while close to the formulation of the single-image super-resolution problem, is in fact rather different. Sing…
Adversarial attacks are small, carefully crafted perturbations, imperceptible to the naked eye; that when added to an image cause deep learning models to misclassify the image with potentially detrimental outcomes. With the rise of artificial intelligence models in consumer safety and security intensive industries such as self-driving cars, camera surveillance and face recognition, there is a growing need for guarding against adversarial attacks. In this paper, we present an approach that uses metamorphic testing principles to automatically detect such adversarial attacks. The approach can det…
Intelligence and Space Research Division（情报与空间研究部）；Descartes Labs, Inc.（笛卡尔实验室公司）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
This paper introduces a new method of generating realistic pervasive changes in the context of evaluating the effectiveness of change detection algorithms in controlled settings. The method, a cycle-consistent adversarial network (CycleGAN), requires low quantities of training data to generate realistic changes. Here we show an application of CycleGAN in creating realistic snow-covered scenes of multispectral Sentinel-2 imagery, and demonstrate how these images can be used as a test bed for anomalous change detection algorithms.
Meta-learning model can quickly adapt to new tasks using few-shot labeled data. However, despite achieving good generalization on few-shot classification tasks, it is still challenging to improve the adversarial robustness of the meta-learning model in few-shot learning. Although adversarial training (AT) methods such as Adversarial Query (AQ) can improve the adversarially robust performance of meta-learning models, AT is still computationally expensive training. On the other hand, meta-learning models trained with AT will drop significant accuracy on the original clean images. This paper prop…
Kyoto University（京都大学）；e-Growth Co., Ltd.（e-Growth 有限公司）；Nara Medical University（奈良县立医科大学）；Rakuwakai Otowa Hospital（洛和会音羽医院）
The reduction of metal artifacts in computed tomography (CT) images, specifically for strong artifacts generated from multiple metal objects, is a challenging issue in medical imaging research. Although there have been some studies on supervised metal artifact reduction through the learning of synthesized artifacts, it is difficult for simulated artifacts to cover the complexity of the real physical phenomena that may be observed in X-ray propagation. In this paper, we introduce metal artifact reduction methods based on an unsupervised volume-to-volume translation learned from clinical CT imag…
Bosch Center for Artificial Intelligence（博世人工智能中心）
Recently demonstrated physical-world adversarial attacks have exposed vulnerabilities in perception systems that pose severe risks for safety-critical applications such as autonomous driving. These attacks place adversarial artifacts in the physical world that indirectly cause the addition of a universal patch to inputs of a model that can fool it in a variety of contexts. Adversarial training is the most effective defense against image-dependent adversarial attacks. However, tailoring adversarial training to universal patches is computationally expensive since the optimal universal patch depe…
Meta-learning provides a promising way for learning to efficiently learn and achieves great success in many applications. However, most meta-learning literature focuses on dealing with tasks from a same domain, making it brittle to generalize to tasks from the other unseen domains. In this work, we address this problem by simulating tasks from the other unseen domains to improve the generalization and robustness of meta-learning method. Specifically, we propose a model-agnostic shift layer to learn how to simulate the domain shift and generate pseudo tasks, and develop a new adversarial learni…
Rensselaer Polytechnic Institute（伦斯勒理工学院）；University of Michigan（密歇根大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；Northeastern University（东北大学）；Michigan State University（密歇根州立大学）
Model-agnostic meta-learning (MAML) has emerged as one of the most successful meta-learning techniques in few-shot learning. It enables us to learn a meta-initialization} of model parameters (that we call meta-model) to rapidly adapt to new tasks using a small amount of labeled training data. Despite the generalization power of the meta-model, it remains elusive that how adversarial robustness can be maintained by MAML in few-shot learning. In addition to generalization, robustness is also desired for a meta-model to defend adversarial examples (attacks). Toward promoting adversarial robustnes…
We propose a novel method for creating adversarial examples. Instead of perturbing pixels, we use an encoder-decoder representation of the input image and perturb intermediate layers in the decoder. This changes the high-level features provided by the generative model. Therefore, our perturbation possesses semantic meaning, such as a longer beak or green tints. We formulate this task as an optimization problem by minimizing the Wasserstein distance between the adversarial and initial images under a misclassification constraint. We employ the projected gradient method with a simple inexact proj…
Beijing Institute of Technology（北京理工大学）；Beijing Institute of Technology Chongqing Innovation Center（北京理工大学重庆创新中心）
Deep neural networks (DNNs) have achieved remarkable success in diverse fields. However, it has been demonstrated that DNNs are very vulnerable to adversarial examples even in black-box settings. A large number of black-box attack methods have been proposed to in the literature. However, those methods usually suffer from low success rates and large query counts, which cannot fully satisfy practical purposes. In this paper, we propose a hybrid attack method which trains meta adversarial perturbations (MAPs) on surrogate models and performs black-box attacks by estimating gradients of the models…
Tsinghua University（清华大学）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Tencent AI Lab（腾讯人工智能实验室）；University of Electronic Science and Technology of China（电子科技大学）
In the scenario of black-box adversarial attack, the target model's parameters are unknown, and the attacker aims to find a successful adversarial perturbation based on query feedback under a query budget. Due to the limited feedback information, existing query-based black-box attack methods often require many queries for attacking each benign example. To reduce query cost, we propose to utilize the feedback information across historical attacks, dubbed example-level adversarial transferability. Specifically, by treating the attack on each benign example as one task, we develop a meta-learning…
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心（BNRist））；JD AI Research（京东人工智能研究院）；Shanghai University（上海大学）
Deep neural networks (DNNs) are vulnerable to adversarial attack which is maliciously implemented by adding human-imperceptible perturbation to images and thus leads to incorrect prediction. Existing studies have proposed various methods to detect the new adversarial attacks. However, new attack methods keep evolving constantly and yield new adversarial examples to bypass the existing detectors. It needs to collect tens of thousands samples to train detectors, while the new attacks evolve much more frequently than the high-cost data collection. Thus, this situation leads the newly evolved atta…
MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；National Yang Ming Chiao Tung University（阳明交通大学）；IBM Research（IBM研究院）
A plethora of attack methods have been proposed to generate adversarial examples, among which the iterative methods have been demonstrated the ability to find a strong attack. However, the computation of an adversarial perturbation for a new data point requires solving a time-consuming optimization problem from scratch. To generate a stronger attack, it normally requires updating a data point with more iterations. In this paper, we show the existence of a meta adversarial perturbation (MAP), a better initialization that causes natural images to be misclassified with high probability after bein…
When creating digital art, coloring and shading are often time consuming tasks that follow the same general patterns. A solution to automatically colorize raw line art would have many practical applications. We propose a setup utilizing two networks in tandem: a color prediction network based only on outlines, and a shading network conditioned on both outlines and a color scheme. We present processing methods to limit information passed in the color scheme, improving generalization. Finally, we demonstrate natural-looking results when colorizing outlines from scratch, as well as from a messy, …
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent（腾讯）
In recent years, research on adversarial attacks has become a hot spot. Although current literature on the transfer-based adversarial attack has achieved promising results for improving the transferability to unseen black-box models, it still leaves a long way to go. Inspired by the idea of meta-learning, this paper proposes a novel architecture called Meta Gradient Adversarial Attack (MGAA), which is plug-and-play and can be integrated with any existing gradient-based attack method for improving the cross-model transferability. Specifically, we randomly sample multiple models from a model zoo…
University of Siegen（锡根大学）；University of Maryland（马里兰大学）
Data poisoning is a threat model in which a malicious actor tampers with training data to manipulate outcomes at inference time. A variety of defenses against this threat model have been proposed, but each suffers from at least one of the following flaws: they are easily overcome by adaptive attacks, they severely reduce testing performance, or they cannot generalize to diverse data poisoning threat models. Adversarial training, and its variants, are currently considered the only empirically strong defense against (inference-time) adversarial attacks. In this work, we extend the adversarial tr…
In this paper, we propose novel generative models for creating adversarial examples, slightly perturbed images resembling natural images but maliciously crafted to fool pre-trained models. We present trainable deep neural networks for transforming images to adversarial perturbations. Our proposed models can produce image-agnostic and image-dependent perturbations for both targeted and non-targeted attacks. We also demonstrate that similar architectures can achieve impressive results in fooling classification and semantic segmentation models, obviating the need for hand-crafting attack methods …
