Realistic fine-grained multi-agent simulation of real-world complex systems is crucial for many downstream tasks such as reinforcement learning. Recent work has used generative models (GANs in particular) for providing high-fidelity simulation of real-world systems. However, such generative models are often monolithic and miss out on modeling the interaction in multi-agent systems. In this work, we take a first step towards building multiple interacting generative models (GANs) that reflects the interaction in real world. We build and analyze a hierarchical set-up where a higher-level GAN is conditioned on the output of multiple lower-level GANs. We present a technique of using feedback from the higher-level GAN to improve performance of lower-level GANs. We mathematically characterize the conditions under which our technique is impactful, including understanding the transfer learning nature of our set-up. We present three distinct experiments on synthetic data, time series data, and image domain, revealing the wide applicability of our technique.
Duke University（杜克大学）；Renmin University of China（中国人民大学）；Facebook, Inc（脸书公司）；IIT Kanpur（坎普尔印度理工学院）
Traditional collaborative filtering (CF) based recommender systems tend to perform poorly when the user-item interactions/ratings are highly scarce. To address this, we propose a learning framework that improves collaborative filtering with a synthetic feedback loop (CF-SFL) to simulate the user feedback. The proposed framework consists of a "recommender" and a "virtual user". The "recommender" is formulated as a CF model, recommending items according to observed user preference. The "virtual user" estimates rewards from the recommended items and generates a \emph{feedback} in addition to the …
Though robot learning is often formulated in terms of discrete-time Markov decision processes (MDPs), physical robots require near-continuous multiscale feedback control. Machines operate on multiple asynchronous sensing modalities, each with different frequencies, e.g., video frames at 30Hz, proprioceptive state at 100Hz, force-torque data at 500Hz, etc. While the classic approach is to batch observations into fixed-time windows then pass them through feed-forward encoders (e.g., with deep networks), we show that there exists a more elegant approach -- one that treats policy learning as model…
ShanghaiTech University（上海科技大学）；University of Notre Dame（圣母大学）；Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Estimation of spatially-varying parameters for computationally expensive forward models governed by partial differential equations is addressed. A novel multiscale Bayesian inference approach is introduced based on deep probabilistic generative models. Such generative models provide a flexible representation by inferring on each scale a low-dimensional latent encoding while allowing hierarchical parameter generation from coarse- to fine-scales. Combining the multiscale generative model with Markov Chain Monte Carlo (MCMC), inference across scales is achieved enabling us to efficiently obtain p…
We propose a Multiscale Invertible Generative Network (MsIGN) and associated training algorithm that leverages multiscale structure to solve high-dimensional Bayesian inference. To address the curse of dimensionality, MsIGN exploits the low-dimensional nature of the posterior, and generates samples from coarse to fine scale (low to high dimension) by iteratively upsampling and refining samples. MsIGN is trained in a multi-stage manner to minimize the Jeffreys divergence, which avoids mode dropping in high-dimensional cases. On two high-dimensional Bayesian inverse problems, we show superior pe…
We propose a MultiScale AutoEncoder(MSAE) based extreme image compression framework to offer visually pleasing reconstruction at a very low bitrate. Our method leverages the "priors" at different resolution scale to improve the compression efficiency, and also employs the generative adversarial network(GAN) with multiscale discriminators to perform the end-to-end trainable rate-distortion optimization. We compare the perceptual quality of our reconstructions with traditional compression algorithms using High-Efficiency Video Coding(HEVC) based Intra Profile and JPEG2000 on the public Cityscape…
The variational autoencoder (VAE) framework remains a popular option for training unsupervised generative models, especially for discrete data where generative adversarial networks (GANs) require workaround to create gradient for the generator. In our work modeling US postal addresses, we show that our discrete VAE with tree recursive architecture demonstrates limited capability of capturing field correlations within structured data, even after overcoming the challenge of posterior collapse with scheduled sampling and tuning of the KL-divergence weight $β$. Worse, VAE seems to have difficulty …
The advancement of the Artificial Intelligence (AI) technologies makes it possible to learn stylistic design criteria from existing maps or other visual art and transfer these styles to make new digital maps. In this paper, we propose a novel framework using AI for map style transfer applicable across multiple map scales. Specifically, we identify and transfer the stylistic elements from a target group of visual examples, including Google Maps, OpenStreetMap, and artistic paintings, to unstylized GIS vector data through two generative adversarial network (GAN) models. We then train a binary cl…
KIST Europe（韩国科学技术研究院欧洲分院）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Korea Institute of Science and Technology Europe Forschungsgesellschaft mbH（韩国科学技术研究院欧洲研究有限公司）
Bearing is a key component in industrial machinery and its failure may lead to unwanted downtime and economic loss. Hence, it is necessary to predict the remaining useful life (RUL) of bearings. Conventional data-driven approaches of RUL prediction require expert domain knowledge for manual feature extraction and may suffer from data distribution discrepancy between training and test data. In this study, we propose a novel generalized multiscale feature extraction method with generative adversarial networks. The adversarial training learns the distribution of training data from different beari…
Scale-Adaptive Generative Flows for Multiscale Scientific Data
Department of Mathematics, University of California, Los Angeles（加州大学洛杉矶分校数学系）；Machine Learning Lab, Capital Fund Management（资本基金管理有限公司机器学习实验室）；Courant Institute, New York University（纽约大学柯朗研究所）
Scale-Aware Adversarial Analysis: A Diagnostic for Generative AI in Multiscale Complex Systems
School of Astronomy and Space Science, Nanjing University（南京大学天文与空间科学学院）；South-Western Institute for Astronomy Research, Yunnan University（云南西南天文研究所）；Key Laboratory of Modern Astronomy and Astrophysics (Nanjing University), Ministry of Education（南京大学现代天文与天体物理重点实验室）；Canadian Institute for Theoretical Astrophysics, University of Toronto（多伦多大学加拿大理论天体物理研究所）
University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Generative modeling of 3D brain MRIs presents difficulties in achieving high visual fidelity while ensuring sufficient coverage of the data distribution. In this work, we propose to address this challenge with composable, multiscale morphological transformations in a variational autoencoder (VAE) framework. These transformations are applied to a chosen reference brain image to generate MRI volumes, equipping the model with strong anatomical inductive biases. We structure the VAE latent space in a way such that the model covers the data distribution sufficiently well. We show substantial perfor…
LyapuFlow: Controlling Generative Flows with Lyapunov Feedback for Inverse Problems
Lawrence Berkeley National Lab（劳伦斯伯克利国家实验室）；Georgia Institute of Technology（佐治亚理工学院）；Siemens（西门子公司）；International Computer Science Institute（国际计算机科学研究所）
Metamaterials are emerging as a new paradigmatic material system to render unprecedented and tailorable properties for a wide variety of engineering applications. However, the inverse design of metamaterial and its multiscale system is challenging due to high-dimensional topological design space, multiple local optima, and high computational cost. To address these hurdles, we propose a novel data-driven metamaterial design framework based on deep generative modeling. A variational autoencoder (VAE) and a regressor for property prediction are simultaneously trained on a large metamaterial datab…
California Institute of Technology（加州理工学院）；NVIDIA（英伟达）；Rice University（莱斯大学）
Neural networks are vulnerable to input perturbations such as additive noise and adversarial attacks. In contrast, human perception is much more robust to such perturbations. The Bayesian brain hypothesis states that human brains use an internal generative model to update the posterior beliefs of the sensory input. This mechanism can be interpreted as a form of self-consistency between the maximum a posteriori (MAP) estimation of an internal generative model and the external environment. Inspired by such hypothesis, we enforce self-consistency in neural networks by incorporating generative rec…
Access to high-quality education at scale is limited by the difficulty of providing student feedback on open-ended assignments in structured domains like computer programming, graphics, and short response questions. This problem has proven to be exceptionally difficult: for humans, it requires large amounts of manual work, and for computers, until recently, achieving anything near human-level accuracy has been unattainable. In this paper, we present generative grading: a novel computational approach for providing feedback at scale that is capable of accurately grading student work and providin…
Multiscale simulations are demanding in terms of computational resources. In the context of continuum micromechanics, the multiscale problem arises from the need of inferring macroscopic material parameters from the microscale. If the underlying microstructure is explicitly given by means of microCT-scans, convolutional neural networks can be used to learn the microstructure-property mapping, which is usually obtained from computational homogenization. The CNN approach provides a significant speedup, especially in the context of heterogeneous or functionally graded materials. Another applicati…
Baylor College of Medicine（贝勒医学院）；Rice University（莱斯大学）
A major obstacle to developing artificial intelligence applications capable of true lifelong learning is that artificial neural networks quickly or catastrophically forget previously learned tasks when trained on a new one. Numerous methods for alleviating catastrophic forgetting are currently being proposed, but differences in evaluation protocols make it difficult to directly compare their performance. To enable more meaningful comparisons, here we identified three distinct scenarios for continual learning based on whether task identity is known and, if it is not, whether it needs to be infe…
Machine learning predictors have been increasingly applied in production settings, including in one of the world's largest hiring platforms, Hired, to provide a better candidate and recruiter experience. The ability to provide actionable feedback is desirable for candidates to improve their chances of achieving success in the marketplace. Until recently, however, methods aimed at providing actionable feedback have been limited in terms of realism and latency. In this work, we demonstrate how, by applying a newly introduced method based on Generative Adversarial Networks (GANs), we are able to …
A Markovian View of Iterative-Feedback Loops in Image Generative Models: Neural Resonance and Model Collapse
Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN, USA（印第安纳大学信息学、计算与工程学院）；Department of Computer Science, Stonehill College, Easton, MA, USA（石溪学院计算机科学系）
Massachusetts Institute of Technology（麻省理工学院）；Google Brain（谷歌大脑）
In the quest towards general artificial intelligence (AI), researchers have explored developing loss functions that act as intrinsic motivators in the absence of external rewards. This paper argues that such research has overlooked an important and useful intrinsic motivator: social interaction. We posit that making an AI agent aware of implicit social feedback from humans can allow for faster learning of more generalizable and useful representations, and could potentially impact AI safety. We collect social feedback in the form of facial expression reactions to samples from Sketch RNN, an LST…
Generative Multi-modal Feedback for Singing Voice Synthesis Evaluation
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Columbia University（哥伦比亚大学）；Dalian University of Technology（大连理工大学）；The Chinese University of Hong Kong（香港中文大学）
Generative Model-Aided Continual Learning for CSI Feedback in FDD mMIMO-OFDM Systems
College of Information Science and Technology, Donghua University（东华大学信息科学与技术学院）；Department of Information and Computer Science, Keio University（Keio大学信息与计算机科学系）；School of Computing and Information Technology, Great Bay University（Great Bay大学计算与信息科技学院）；Great Bay Institute for Advanced Study (GBIAS)（Great Bay先进研究 institutes）；Department of Engineering, Nottingham Trent University（Nottingham Trent大学工程系）
