State-of-the-art deep learning methods have shown a remarkable capacity to model complex data domains, but struggle with geospatial data. In this paper, we introduce SpaceGAN, a novel generative model for geospatial domains that learns neighbourhood structures through spatial conditioning. We propose to enhance spatial representation beyond mere spatial coordinates, by conditioning each data point on feature vectors of its spatial neighbours, thus allowing for a more flexible representation of the spatial structure. To overcome issues of training convergence, we employ a metric capturing the loss in local spatial autocorrelation between real and generated data as stopping criterion for SpaceGAN parametrization. This way, we ensure that the generator produces synthetic samples faithful to the spatial patterns observed in the input. SpaceGAN is successfully applied for data augmentation and outperforms compared to other methods of synthetic spatial data generation. Finally, we propose an ensemble learning framework for the geospatial domain, taking augmented SpaceGAN samples as training data for a set of ensemble learners. We empirically show the superiority of this approach over conventional ensemble learning approaches and rivaling spatial data augmentation methods, using synthetic and real-world prediction tasks. Our findings suggest that SpaceGAN can be used as a tool for (1) artificially inflating sparse geospatial data and (2) improving generalization of geospatial models.
Stanford University（斯坦福大学）；California Institute of Technology（加州理工学院）；Cornell University（康奈尔大学）
A challenge in training discriminative models like neural networks is obtaining enough labeled training data. Recent approaches use generative models to combine weak supervision sources, like user-defined heuristics or knowledge bases, to label training data. Prior work has explored learning accuracies for these sources even without ground truth labels, but they assume that a single accuracy parameter is sufficient to model the behavior of these sources over the entire training set. In particular, they fail to model latent subsets in the training data in which the supervision sources perform d…
IBM Research, T.J. Watson Research Center（IBM研究院托马斯·J·沃森研究中心）
Deep generative models have emerged as a powerful tool for learning useful molecular representations and designing novel molecules with desired properties, with applications in drug discovery and material design. However, most existing deep generative models are restricted due to lack of spatial information. Here we propose augmentation of deep generative models with topological data analysis (TDA) representations, known as persistence images, for robust encoding of 3D molecular geometry. We show that the TDA augmentation of a character-based Variational Auto-Encoder (VAE) outperforms state-of…
Augmenting generative models with biomedical knowledge graphs improves targeted drug discovery
Edwardson School of Industrial Engineering and the Institute of Cancer Research, Purdue University（埃德沃兹工业工程学院和癌症研究所，普渡大学）；Elmore Family School of Electrical and Computer Engineering, Purdue University（埃尔摩家庭电气与计算机工程学院，普渡大学）
Beyond 9-to-5: A Generative Model for Augmenting Mobility Data of Underrepresented Shift Workers
Department of Civil and Environmental Engineering, Samueli School of Engineering, University of California at Los Angeles（土木与环境工程系，工程学院，加州大学洛杉矶分校）；Novateur Research Solutions（Novateur研究解决方案）
针对轮班工作者在交通调查中代表性不足的问题，提出基于transformer的GPS轨迹生成方法，利用周期感知嵌入与转移损失补全其活动模式，并在Los Angeles County数据上取得平均JSD<0.02的分布对齐效果。
Henry Ford Health System（亨利福特医疗系统）；Wayne State University（韦恩州立大学）
Every year thousands of patients are diagnosed with a glioma, a type of malignant brain tumor. Physicians use MR images as a key tool in the diagnosis and treatment of these patients. Neural networks show great potential to aid physicians in the medical image analysis. This study investigates the use of varying amounts of synthetic brain T1-weighted (T1), post-contrast T1-weighted (T1Gd), T2-weighted (T2), and T2 Fluid Attenuated Inversion Recovery (FLAIR) MR images created by a generative adversarial network to overcome the lack of annotated medical image data in training separate 2D U-Nets t…
A Research Roadmap for Augmenting Software Engineering Processes and Software Products with Generative AI
University of Naples "Federico II" Napoli Italy；Ruhr Institute for Software Technology (paluno), University of Duisburg-Essen Essen Germany；Karlsruhe Institute of Technology (KIT) Karlsruhe Germany；Gran Sasso Science Institute (GSSI) L'Aquila Italy；University of Naples "Federico II"；Ruhr Institute for Software Technology (paluno), University of Duisburg-Essen；Karlsruhe Institute of Technology (KIT)；Gran Sasso Science Institute (GSSI)
Many mathematical optimization algorithms fail to sufficiently explore the solution space of high-dimensional nonlinear optimization problems due to the curse of dimensionality. This paper proposes generative models as a complement to optimization algorithms to improve performance in problems with high dimensionality. To demonstrate this method, a conditional generative adversarial network (C-GAN) is used to augment the solutions produced by a genetic algorithm (GA) for a 311-dimensional nonconvex multi-objective mixed-integer nonlinear optimization. The C-GAN, composed of two networks with th…
We introduce a self-attending task generative adversarial network (SATGAN) and apply it to the problem of augmenting synthetic high contrast scientific imagery of resident space objects with realistic noise patterns and sensor characteristics learned from collected data. Augmenting these synthetic data is challenging due to the highly localized nature of semantic content in the data that must be preserved. Real collected images are used to train a network what a given class of sensor's images should look like. The trained network then acts as a filter on noiseless context images and outputs re…
Generative flows are promising tractable models for density modeling that define probabilistic distributions with invertible transformations. However, tractability imposes architectural constraints on generative flows, making them less expressive than other types of generative models. In this work, we study a previously overlooked constraint that all the intermediate representations must have the same dimensionality with the original data due to invertibility, limiting the width of the network. We tackle this constraint by augmenting the data with some extra dimensions and jointly learning a g…
Synthetic Data Generation for Augmenting Small Samples
CHEO Research Institute（CHEO研究所）；University of Ottawa（渥太华大学）；Charité - Universitaetsmedizin Berlin（柏林夏里特医学院）；Hospital for Sick Children（病童医院）；Ottawa Hospital Research Institute（渥太华医院研究所）；McMaster University（麦克马斯特大学）；OpenSourceResearch
Invariances to translation, rotation and other spatial transformations are a hallmark of the laws of motion, and have widespread use in the natural sciences to reduce the dimensionality of systems of equations. In supervised learning, such as in image classification tasks, rotation, translation and scale invariances are used to augment training datasets. In this work, we use data augmentation in a similar way, exploiting symmetry in the quadruped domain of the DeepMind control suite (Tassa et al. 2018) to add to the trajectories experienced by the actor in the actor-critic algorithm of Abdolma…
We develop a new framework for learning variational autoencoders and other deep generative models that balances generative and discriminative goals. Our framework optimizes model parameters to maximize a variational lower bound on the likelihood of observed data, subject to a task-specific prediction constraint that prevents model misspecification from leading to inaccurate predictions. We further enforce a consistency constraint, derived naturally from the generative model, that requires predictions on reconstructed data to match those on the original data. We show that these two contribution…
Vector Institute for Artificial Intelligence（向量人工智能研究所）；University of Guelph（圭尔夫大学）
Although machine learning (ML) has been successful in automating various software engineering needs, software testing still remains a highly challenging topic. In this paper, we aim to improve the generative testing of software by directly augmenting the random number generator (RNG) with a deep reinforcement learning (RL) agent using an efficient, automatically extractable state representation of the software under test. Using the Cosmos SDK as the testbed, we show that the proposed DeepRNG framework provides a statistically significant improvement to the testing of the highly complex softwar…
University of Oslo（奥斯陆大学）；National University of Singapore（新加坡国立大学）
Supervised machine learning applications in the health domain often face the problem of insufficient training datasets. The quantity of labelled data is small due to privacy concerns and the cost of data acquisition and labelling by a medical expert. Furthermore, it is quite common that collected data are unbalanced and getting enough data to personalize models for individuals is very expensive or even infeasible. This paper addresses these problems by (1) designing a recurrent Generative Adversarial Network to generate realistic synthetic data and to augment the original dataset, (2) enabling…
TU Dresden（德累斯顿工业大学）；GE Global Research（通用电气全球研究中心）
Microstructure reconstruction is an important and emerging field of research and an essential foundation to improving inverse computational materials engineering (ICME). Much of the recent progress in the field is made based on generative adversarial networks (GANs). Although excellent results have been achieved throughout a variety of materials, challenges remain regarding the interpretability of the model's latent space as well as the applicability to extremely small data sets. The present work addresses these issues by introducing DA-VEGAN, a model with two central innovations. First, a $β$…
Logic-RAG: Augmenting Large Multimodal Models with Visual-Spatial Knowledge for Road Scene Understanding
College of Information Sciences and Technology, Pennsylvania State University（宾夕法尼亚州立大学信息科学与技术学院）；Department of Mathematics and Computer Science, Drake University（德雷克大学数学与计算机科学系）
本文提出 Logic-RAG 框架，通过一阶逻辑动态知识库与逻辑推理增强 LMM 的道路场景空间理解，显著提升 GPT-4V 和 Claude 3.5 的问答准确率。
University of Toronto（多伦多大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Vector Institute for Artificial Intelligence（矢量人工智能研究所）；Canadian Institute for Advanced Research (CIFAR)（加拿大高等研究院（CIFAR））
Challenges in natural sciences can often be phrased as optimization problems. Machine learning techniques have recently been applied to solve such problems. One example in chemistry is the design of tailor-made organic materials and molecules, which requires efficient methods to explore the chemical space. We present a genetic algorithm (GA) that is enhanced with a neural network (DNN) based discriminator model to improve the diversity of generated molecules and at the same time steer the GA. We show that our algorithm outperforms other generative models in optimization tasks. We furthermore p…
Augmenting medical image classifiers with synthetic data from latent diffusion models
Harvard Medical School（哈佛医学院）；University of Oxford（牛津大学）；Northwestern University（西北大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）；Stanford School of Medicine（斯坦福医学院）
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Heriot-Watt University（赫瑞-瓦特大学）；BASF Corporation（巴斯夫公司）
Nigam et al. reported a genetic algorithm (GA) utilizing the SELFIES representation and also propose an adaptive, neural network-based penalty that is supposed to improve the diversity of the generated molecules. The main claims of the paper are that this GA outperforms other generative techniques (as measured by the penalized logP) and that a neural network-based adaptive penalty increases the diversity of the generated molecules. In this work, we investigated the reproducibility of their claims. Overall, we were able to reproduce comparable results using the SELFIES-based GA, but mostly by e…
Astronomy of the 21st century increasingly finds itself with extreme quantities of data. This growth in data is ripe for modern technologies such as deep image processing, which has the potential to allow astronomers to automatically identify, classify, segment and deblend various astronomical objects. In this paper, we explore the use of chained generative adversarial networks (GANs), a class of generative models that learn mappings from latent spaces to data distributions by modelling the joint distribution of the data, to produce physically realistic galaxy images as one use case of such mo…
Generative AI like ChatGPT in Blockchain Federated Learning: use cases, opportunities and future
Southern Illinois University（南伊利诺伊大学）；University of Asia Pacific（亚太大学）；Green University of Bangladesh（孟加拉国绿色大学）；University of North Texas（北得克萨斯大学）；John Hopkins University（约翰霍普金斯大学）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；University of Pittsburgh（匹兹堡大学）
Detecting earthquake events from seismic time series has proved itself a challenging task. Manual detection can be expensive and tedious due to the intensive labor and large scale data set. In recent years, automatic detection methods based on machine learning have been developed to improve accuracy and efficiency. However, the accuracy of those methods relies on a sufficient amount of high-quality training data, which itself can be expensive to obtain due to the requirement of domain knowledge and subject matter expertise. This paper is to resolve this dilemma by answering two questions: (1) …
A Survey of Generative Techniques for Spatial-Temporal Data Mining
University of Hong Kong（香港大学）；Peking University（北京大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）；Westlake University（西湖大学）；University of Queensland（昆士兰大学）；Huawei Cloud & AI（华为云与人工智能）；Aalborg University（奥尔堡大学）
Latent-Constrained Conditional VAEs for Augmenting Large-Scale Climate Ensembles
Hong Kong Polytechnic University（香港理工大学）；Georgia Institute of Technology（佐治亚理工学院）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；TomTom North America Inc.（TomTom北美公司）
University of Cambridge（剑桥大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Microsoft Research（微软研究院）；DeepMind
Traditional deep generative models of images and other spatial modalities can only generate fixed sized outputs. The generated images have exactly the same resolution as the training images, which is dictated by the number of layers in the underlying neural network. Recent work has shown, however, that feeding spatial noise vectors into a fully convolutional neural network enables both generation of arbitrary resolution output images as well as training on arbitrary resolution training images. While this work has provided impressive empirical results, little theoretical interpretation was prov…
Fondazione Bruno Kessler Research Institute（布鲁诺·凯斯勒基金会研究院）；University of Trento（特伦托大学）；University of Manchester（曼彻斯特大学）
Several approaches have been developed to mitigate algorithmic bias stemming from health data poverty, where minority groups are underrepresented in training datasets. Augmenting the minority class using resampling (such as SMOTE) is a widely used approach due to the simplicity of the algorithms. However, these algorithms decrease data variability and may introduce correlations between samples, giving rise to the use of generative approaches based on GAN. Generation of high-dimensional, time-series, authentic data that provides a wide distribution coverage of the real data, remains a challengi…
ETH Zurich（苏黎世联邦理工学院）；Swiss AI Lab IDSIA, USI（瑞士IDSIA人工智能实验室（美国信息研究所））；Max-Planck Institute（马克斯·普朗克研究所）
How to improve generative modeling by better exploiting spatial regularities and coherence in images? We introduce a novel neural network for building image generators (decoders) and apply it to variational autoencoders (VAEs). In our spatial dependency networks (SDNs), feature maps at each level of a deep neural net are computed in a spatially coherent way, using a sequential gating-based mechanism that distributes contextual information across 2-D space. We show that augmenting the decoder of a hierarchical VAE by spatial dependency layers considerably improves density estimation over baseli…
Imperial College London（帝国理工学院）；Accenture Labs（埃森哲实验室）
Class imbalance can often degrade predictive performance of supervised learning algorithms. Balanced classes can be obtained by oversampling exact copies, with noise, or interpolation between nearest neighbours (as in traditional SMOTE methods). Oversampling tabular data using augmentation, as is typical in computer vision tasks, can be achieved with deep generative models. Deep generative models are effective data synthesisers due to their ability to capture complex underlying distributions. Synthetic data in healthcare can enhance interoperability between healthcare providers by ensuring pat…
Technical University of Denmark（丹麦技术大学）；DeepMind（深度思考）
In model-based reinforcement learning, generative and temporal models of environments can be leveraged to boost agent performance, either by tuning the agent's representations during training or via use as part of an explicit planning mechanism. However, their application in practice has been limited to simplistic environments, due to the difficulty of training such models in larger, potentially partially-observed and 3D environments. In this work we introduce a novel action-conditioned generative model of such challenging environments. The model features a non-parametric spatial memory system…
A machine learning model, under the influence of observed or unobserved confounders in the training data, can learn spurious correlations and fail to generalize when deployed. For image classifiers, augmenting a training dataset using counterfactual examples has been empirically shown to break spurious correlations. However, the counterfactual generation task itself becomes more difficult as the level of confounding increases. Existing methods for counterfactual generation under confounding consider a fixed set of interventions (e.g., texture, rotation) and are not flexible enough to capture d…
Generative adversary networks (GANs) suffer from training pathologies such as instability and mode collapse, which mainly arise from a lack of diversity in their adversarial interactions. Co-evolutionary GAN (CoE-GAN) training algorithms have shown to be resilient to these pathologies. This article introduces Mustangs, a spatially distributed CoE-GAN, which fosters diversity by using different loss functions during the training. Experimental analysis on MNIST and CelebA demonstrated that Mustangs trains statistically more accurate generators.
Crowdsourcing provides an efficient label collection schema for supervised machine learning. However, to control annotation cost, each instance in the crowdsourced data is typically annotated by a small number of annotators. This creates a sparsity issue and limits the quality of machine learning models trained on such data. In this paper, we study how to handle sparsity in crowdsourced data using data augmentation. Specifically, we propose to directly learn a classifier by augmenting the raw sparse annotations. We implement two principles of high-quality augmentation using Generative Adversar…
Modeling the distribution of natural images is challenging, partly because of strong statistical dependencies which can extend over hundreds of pixels. Recurrent neural networks have been successful in capturing long-range dependencies in a number of problems but only recently have found their way into generative image models. We here introduce a recurrent image model based on multi-dimensional long short-term memory units which are particularly suited for image modeling due to their spatial structure. Our model scales to images of arbitrary size and its likelihood is computationally tractable…
National Research Council Canada（加拿大国家研究委员会）；University of Ottawa（渥太华大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Brock University（布鲁克大学）
Deploying generative machine learning techniques to generate novel chemical structures based on molecular fingerprint representation has been well established in molecular design. Typically, sequential learning (SL) schemes such as hidden Markov models (HMM) and, more recently, in the sequential deep learning context, recurrent neural network (RNN) and long short-term memory (LSTM) were used extensively as generative models to discover unprecedented molecules. To this end, emission probability between two states of atoms plays a central role without considering specific chemical or physical pr…
Hybrid Adversarial Spectral Loss Conditional Generative Adversarial Networks for Signal Data Augmentation in Ultra-precision Machining Surface Roughness Prediction
KU Leuven（鲁汶大学）；Leuven Brain Institute（鲁汶脑科学研究所）；Maastricht UMC+（马斯特里赫特大学医学中心+）；University Hospitals Leuven（鲁汶大学医院）；Delft University of Technology（代尔夫特理工大学）
EEG-correlated fMRI analysis is widely used to detect regional blood oxygen level dependent fluctuations that are significantly synchronized to interictal epileptic discharges, which can provide evidence for localizing the ictal onset zone. However, such an asymmetrical, mass-univariate approach cannot capture the inherent, higher order structure in the EEG data, nor multivariate relations in the fMRI data, and it is nontrivial to accurately handle varying neurovascular coupling over patients and brain regions. We aim to overcome these drawbacks in a data-driven manner by means of a novel stru…
Mythic Inc.（Mythic公司）；Freeus, LLC.（Freeus有限责任公司）；University of Rochester（罗切斯特大学）
Deep neural networks can form high-level hierarchical representations of input data. Various researchers have demonstrated that these representations can be used to enable a variety of useful applications. However, such representations are typically based on the statistics within the data, and may not conform with the semantic representation that may be necessitated by the application. Conditional models are typically used to overcome this challenge, but they require large annotated datasets which are difficult to come by and costly to create. In this paper, we show that semantically-aligned r…
University of Oxford（牛津大学）；Bartlett School of Architecture, UCL（伦敦大学学院巴特莱特建筑学院）
With this work, we investigate the use of Reinforcement Learning (RL) for the generation of spatial assemblies, by combining ideas from Procedural Generation algorithms (Wave Function Collapse algorithm (WFC)) and RL for Game Solving. WFC is a Generative Design algorithm, inspired by Constraint Solving. In WFC, one defines a set of tiles/blocks and constraints and the algorithm generates an assembly that satisfies these constraints. Casting the problem of generation of spatial assemblies as a Markov Decision Process whose states transitions are defined by WFC, we propose an algorithm that uses…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This paper presents a novel graph convolutional neural network (GCNN)-based approach for improving the diagnosis of neurological diseases using scalp-electroencephalograms (EEGs). Although EEG is one of the main tests used for neurological-disease diagnosis, the sensitivity of EEG-based expert visual diagnosis remains at $\sim$50\%. This indicates a clear need for advanced methodology to reduce the false negative rate in detecting abnormal scalp-EEGs. In that context, we focus on the problem of distinguishing the abnormal scalp EEGs of patients with neurological diseases, which were originally…
University of California, Los Angeles (UCLA)（加州大学洛杉矶分校）
Video sequences contain rich dynamic patterns, such as dynamic texture patterns that exhibit stationarity in the temporal domain, and action patterns that are non-stationary in either spatial or temporal domain. We show that a spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns. The model defines a probability distribution on the video sequence, and the log probability is defined by a spatial-temporal ConvNet that consists of multiple layers of spatial-temporal filters to capture spatial-temporal patterns of different scales. The model can be learned from t…
The difficulty in obtaining labeled data relevant to a given task is among the most common and well-known practical obstacles to applying deep learning techniques to new or even slightly modified domains. The data volumes required by the current generation of supervised learning algorithms typically far exceed what a human needs to learn and complete a given task. We investigate ways to expand a given labeled corpus of remote sensed imagery into a larger corpus using Generative Adversarial Networks (GANs). We then measure how these additional synthetic data affect supervised machine learning p…
Deep metric learning aims to learn an embedding space where the distance between data reflects their class equivalence, even when their classes are unseen during training. However, the limited number of classes available in training precludes generalization of the learned embedding space. Motivated by this, we introduce a new data augmentation approach that synthesizes novel classes and their embedding vectors. Our approach can provide rich semantic information to an embedding model and improve its generalization by augmenting training data with novel classes unavailable in the original data. …
Seoul National University（首尔国立大学）；Samsung SDS（三星SDS）
Recent image generation models show remarkable generation performance. However, they mirror strong location preference in datasets, which we call spatial bias. Therefore, generators render poor samples at unseen locations and scales. We argue that the generators rely on their implicit positional encoding to render spatial content. From our observations, the generator's implicit positional encoding is translation-variant, making the generator spatially biased. To address this issue, we propose injecting explicit positional encoding at each scale of the generator. By learning the spatially unbia…
