The advancement of distributed generation technologies in modern power systems has led to a widespread integration of renewable power generation at customer side. However, the intermittent nature of renewable energy poses new challenges to the network operational planning with underlying uncertainties. This paper proposes a novel Bayesian probabilistic technique for forecasting renewable solar generation by addressing data and model uncertainties by integrating bidirectional long short-term memory (BiLSTM) neural networks while compressing the weight parameters using variational autoencoder (VAE). Existing Bayesian deep learning methods suffer from high computational complexities as they require to draw a large number of samples from weight parameters expressed in the form of probability distributions. The proposed method can deal with uncertainty present in model and data in a more computationally efficient manner by reducing the dimensionality of model parameters. The proposed method is evaluated using quantile loss, reconstruction error, and deterministic forecasting evaluation metrics such as root-mean square error. It is inferred from the numerical results that VAE-Bayesian BiLSTM outperforms other probabilistic and deterministic deep learning methods for solar power forecasting in terms of accuracy and computational efficiency for different sizes of the dataset.
When training a variational autoencoder (VAE) on a given dataset, determining the optimal number of latent variables is mostly done by grid search: a costly process in terms of computational time and carbon footprint. In this paper, we explore the intrinsic dimension estimation (IDE) of the data and latent representations learned by VAEs. We show that the discrepancies between the IDE of the mean and sampled representations of a VAE after only a few steps of training reveal the presence of passive variables in the latent space, which, in well-behaved VAEs, indicates a superfluous number of dim…
When designing variational autoencoders (VAEs) or other types of latent space models, the dimensionality of the latent space is typically defined upfront. In this process, it is possible that the number of dimensions is under- or overprovisioned for the application at hand. In case the dimensionality is not predefined, this parameter is usually determined using time- and resource-consuming cross-validation. For these reasons we have developed a technique to shrink the latent space dimensionality of VAEs automatically and on-the-fly during training using Generalized ELBO with Constrained Optimi…
In this work, we have proposed a generative model, called VAE-KRnet, for density estimation or approximation, which combines the canonical variational autoencoder (VAE) with our recently developed flow-based generative model, called KRnet. VAE is used as a dimension reduction technique to capture the latent space, and KRnet is used to model the distribution of the latent variable. Using a linear model between the data and the latent variable, we show that VAE-KRnet can be more effective and robust than the canonical VAE. VAE-KRnet can be used as a density model to approximate either data distr…
ETH Zürich（苏黎世联邦理工学院）；Yandex（俄罗斯搜索引擎公司）；UC Irvine（加州大学欧文分校）
Multivariate time series with missing values are common in areas such as healthcare and finance, and have grown in number and complexity over the years. This raises the question whether deep learning methodologies can outperform classical data imputation methods in this domain. However, naive applications of deep learning fall short in giving reliable confidence estimates and lack interpretability. We propose a new deep sequential latent variable model for dimensionality reduction and data imputation. Our modeling assumption is simple and interpretable: the high dimensional time series has a l…
Forecasting-based Biomedical Time-series Data Synthesis for Open Data and Robust AI
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea（韩国成均馆大学电气工程学院）；Department of Applied AI, Hansung University, Seoul, Republic of Korea（汉城市立大学应用人工智能系）
Duke University（杜克大学）；The Statistical and Applied Mathematical Sciences Institute（统计与应用数学科学研究所）；Oak Ridge National Lab（橡树岭国家实验室）；University of Tennessee（田纳西大学）
In conducting non-linear dimensionality reduction and feature learning, it is common to suppose that the data lie near a lower-dimensional manifold. A class of model-based approaches for such problems includes latent variables in an unknown non-linear regression function; this includes Gaussian process latent variable models and variational auto-encoders (VAEs) as special cases. VAEs are artificial neural networks (ANNs) that employ approximations to make computation tractable; however, current implementations lack adequate uncertainty quantification in estimating the parameters, predictive de…
The paper presents the application of Variational Autoencoders (VAE) for data dimensionality reduction and explorative analysis of mass spectrometry imaging data (MSI). The results confirm that VAEs are capable of detecting the patterns associated with the different tissue sub-types with performance than standard approaches.
The OECD pointed out that the best way to keep students up to school is to intervene as early as possible [1]. Using education big data and deep learning to predict student's score provides new resources and perspectives for early intervention. Previous forecasting schemes often requires manual filter of features , a large amount of prior knowledge and expert knowledge. Deep learning can automatically extract features without manual intervention to achieve better predictive performance. In this paper, the graph neural network matrix filling model (Graph-VAE) based on deep learning can automati…
Hosei University（法政大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）
Content-based image retrieval (CBIR) systems are an emerging technology that supports reading and interpreting medical images. Since 3D brain MR images are high dimensional, dimensionality reduction is necessary for CBIR using machine learning techniques. In addition, for a reliable CBIR system, each dimension in the resulting low-dimensional representation must be associated with a neurologically interpretable region. We propose a localized variational autoencoder (Loc-VAE) that provides neuroanatomically interpretable low-dimensional representation from 3D brain MR images for clinical CBIR. …
Learning suitable latent representations for observed, high-dimensional data is an important research topic underlying many recent advances in machine learning. While traditionally the Gaussian normal distribution has been the go-to latent parameterization, recently a variety of works have successfully proposed the use of manifold-valued latents. In one such work (Davidson et al., 2018), the authors empirically show the potential benefits of using a hyperspherical von Mises-Fisher (vMF) distribution in low dimensionality. However, due to the unique distributional form of the vMF, expressivity …
A Multi-Level Causal Intervention Framework for Mechanistic Interpretability in Variational Autoencoders
Indian Institute of Technology, Patna（印度理工学院巴特那分校）；Rajarshi School of Management Technology, Varanasi（拉贾尔希管理技术学院瓦拉纳西分校）；Jaypee Institute of Information Technology, Noida（杰皮信息技术学院诺伊达分校）
University of Oxford（牛津大学）；Alan Turing Institute（艾伦·图灵研究所）；University of Birmingham（伯明翰大学）；University of Manchester（曼彻斯特大学）
Variational Autoencoders (VAEs) provide a flexible and scalable framework for non-linear dimensionality reduction. However, in application domains such as genomics where data sets are typically tabular and high-dimensional, a black-box approach to dimensionality reduction does not provide sufficient insights. Common data analysis workflows additionally use clustering techniques to identify groups of similar features. This usually leads to a two-stage process, however, it would be desirable to construct a joint modelling framework for simultaneous dimensionality reduction and clustering of feat…
TimeAutoDiff: A Unified Framework for Generation, Imputation, Forecasting, and Time-Varying Metadata Conditioning of Heterogeneous Time Series Tabular Data
The task of predicting stochastic behaviors of road agents in diverse environments is a challenging problem for autonomous driving. To best understand scene contexts and produce diverse possible future states of the road agents adaptively in different environments, a prediction model should be probabilistic, multi-modal, context-driven, and general. We present Conditionalizing Variational AutoEncoders via Hypernetworks (CVAE-H); a conditional VAE that extensively leverages hypernetwork and performs generative tasks for high-dimensional problems like the prediction task. We first evaluate CVAE-…
Utilizing VQ-VAE for End-to-End Health Indicator Generation in Predicting Rolling Bearing RUL
School of Automation, Guangdong University of Petrochemical Technology（广东石油化工学院自动化学院）；School of Automation, Guangdong University of Technology（广东工业大学自动化学院）；Guangdong Provincial Key Laboratory of Petrochemical Equipment and Fault Diagnosis, Guangdong University of Petrochemical Technology（广东石油化工学院广东省石化装备故障诊断重点实验室）
A shift-invariant variational autoencoder (shift-VAE) is developed as an unsupervised method for the analysis of spectral data in the presence of shifts along the parameter axis, disentangling the physically-relevant shifts from other latent variables. Using synthetic data sets, we show that the shift-VAE latent variables closely match the ground truth parameters. The shift VAE is extended towards the analysis of band-excitation piezoresponse force microscopy (BE-PFM) data, disentangling the resonance frequency shifts from the peak shape parameters in a model-free unsupervised manner. The exte…
University of Cambridge（剑桥大学）；Monash University（莫纳什大学）；Imperial College London（帝国理工学院）
It has been long known that sparsity is an effective inductive bias for learning efficient representation of data in vectors with fixed dimensionality, and it has been explored in many areas of representation learning. Of particular interest to this work is the investigation of the sparsity within the VAE framework which has been explored a lot in the image domain, but has been lacking even a basic level of exploration in NLP. Additionally, NLP is also lagging behind in terms of learning sparse representations of large units of text e.g., sentences. We use the VAEs that induce sparse latent re…
The ability to forecast a set of likely yet diverse possible future behaviors of an agent (e.g., future trajectories of a pedestrian) is essential for safety-critical perception systems (e.g., autonomous vehicles). In particular, a set of possible future behaviors generated by the system must be diverse to account for all possible outcomes in order to take necessary safety precautions. It is not sufficient to maintain a set of the most likely future outcomes because the set may only contain perturbations of a single outcome. While generative models such as variational autoencoders (VAEs) have …
Machine learning applied to generate data-driven models are lacking of transparency leading the process engineer to lose confidence in relying on the model predictions to optimize his industrial process. Bringing processes in the industry to a certain level of autonomy using data-driven models is particularly challenging as the first user of those models, is the expert in the process with often decades of experience. It is necessary to expose to the process engineer, not solely the model predictions, but also their interpretability. To that end, several approaches have been proposed in the lit…
ChronoVAE-HOPE: Beyond Attention -- A Next-Generation VAE Foundation Model for Specialized Time Series Classification
Department of Computer Science and Artificial Intelligence（计算机科学与人工智能系）；DiCITS；iMUDS；DaSCI；University of Granada（格拉纳达大学）；Advanced Medical Imaging Group（先进医学成像组）；Instituto de Investigación Biosanitaria de Granada（格拉纳达生物医学研究 institute）；Department of Software Engineering（软件工程系）；Department of Rural Engineering（农村工程系）；University of Córdoba（科尔多瓦大学）
In recent years, the task of video prediction-forecasting future video given past video frames-has attracted attention in the research community. In this paper we propose a novel approach to this problem with Vector Quantized Variational AutoEncoders (VQ-VAE). With VQ-VAE we compress high-resolution videos into a hierarchical set of multi-scale discrete latent variables. Compared to pixels, this compressed latent space has dramatically reduced dimensionality, allowing us to apply scalable autoregressive generative models to predict video. In contrast to previous work that has largely emphasize…
$β$-Variational autoencoders and transformers for reduced-order modelling of fluid flows
Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Spanish National Institute for Aerospace Technology (INTA)（西班牙国家航空航天技术研究院）；Illinois Institute of Technology（伊利诺伊理工学院）；KTH Royal Institute of Technology（瑞典皇家理工学院）
SLAC National Accelerator Laboratory（SLAC国家加速器实验室）
I present a Variational Autoencoder (VAE) trained on collider physics data (specifically boosted $W$ jets), with reconstruction error given by an approximation to the Earth Movers Distance (EMD) between input and output jets. This VAE learns a concrete representation of the data manifold, with semantically meaningful and interpretable latent space directions which are hierarchically organized in terms of their relation to physical EMD scales in the underlying physical generative process. The variation of the latent space structure with a resolution hyperparameter provides insight into scale de…
University of California at Irvine（加州大学欧文分校）；Columbia University（哥伦比亚大学）
While cloud-resolving models can explicitly simulate the details of small-scale storm formation and morphology, these details are often ignored by climate models for lack of computational resources. Here, we explore the potential of generative modeling to cheaply recreate small-scale storms by designing and implementing a Variational Autoencoder (VAE) that performs structural replication, dimensionality reduction, and clustering of high-resolution vertical velocity fields. Trained on ~6*10^6 samples spanning the globe, the VAE successfully reconstructs the spatial structure of convection, perf…
Institute for Advanced Study（高等研究院）；Tsinghua University（清华大学）；University of Cambridge（剑桥大学）；Microsoft Research（微软研究院）
Variational autoencoders (VAE) represent a popular, flexible form of deep generative model that can be stochastically fit to samples from a given random process using an information-theoretic variational bound on the true underlying distribution. Once so-obtained, the model can be putatively used to generate new samples from this distribution, or to provide a low-dimensional latent representation of existing samples. While quite effective in numerous application domains, certain important mechanisms which govern the behavior of the VAE are obfuscated by the intractable integrals and resulting …
Stabilization Analysis and Mode Recognition of Kerosene Supersonic Combustion: A Deep Learning Approach Based on Res-CNN-beta-VAE
City University of Hong Kong（香港城市大学）；State Key Laboratory of High Temperature Gas Dynamics, Institute of Mechanics, Chinese Academy of Sciences（中国科学院力学研究所高温气体动力学国家重点实验室）
Explainable AI for Multivariate Time Series Pattern Exploration: Latent Space Visual Analytics with Temporal Fusion Transformer and Variational Autoencoders in Power Grid Event Diagnosis
DAWP: A framework for global observation forecasting via Data Assimilation and Weather Prediction in satellite observation space
Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong Shanghai AI Laboratory（香港中文大学上海人工智能实验室）；Shanghai AI Laboratory（上海人工智能实验室）
Interpretable Spectral Variational AutoEncoder (ISVAE) for time series clustering
Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Instituto de Investigación Sanitaria Gregorio Marañón（格雷戈里奥·马拉尼翁卫生研究所）；Evidence Based Behaviour (eB2)（循证行为（eB2））；Oxford-Man Institute of Quantitative Finance, University of Oxford（牛津大学牛津-曼氏定量金融研究所）
Osaka University（大阪大学）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Yunnan University（云南大学）
Defining and separating cancer subtypes is essential for facilitating personalized therapy modality and prognosis of patients. The definition of subtypes has been constantly recalibrated as a result of our deepened understanding. During this recalibration, researchers often rely on clustering of cancer data to provide an intuitive visual reference that could reveal the intrinsic characteristics of subtypes. The data being clustered are often omics data such as transcriptomics that have strong correlations to the underlying biological mechanism. However, while existing studies have shown promis…
Variational Autoencoder for Generating Broader-Spectrum prior Proposals in Markov chain Monte Carlo Methods
Department One（部门一）；Department Two（部门二）；National Laboratory for Scientific Computing（国家科学计算实验室）；Department of Mathematical Sciences, The University of Texas at Dallas（数学科学系，德克萨斯大学达拉斯分校）
Max Planck Institute for Polymer Research（马克斯·普朗克聚合物研究所）；Koc University（科奇大学）
Extracting insight from the enormous quantity of data generated from molecular simulations requires the identification of a small number of collective variables whose corresponding low-dimensional free-energy landscape retains the essential features of the underlying system. Data-driven techniques provide a systematic route to constructing this landscape, without the need for extensive a priori intuition into the relevant driving forces. In particular, autoencoders are powerful tools for dimensionality reduction, as they naturally force an information bottleneck and, thereby, a low-dimensional…
$Γ$-VAE: Curvature regularized variational autoencoders for uncovering emergent low dimensional geometric structure in high dimensional data
Cornell University（康奈尔大学）；Sorbonne Université（索邦大学）；Institut Curie Research Center（居里研究所）；PSL University（巴黎文理研究大学）；Inserm（法国国家健康与医学研究院）；Center for Advanced Computing, Cornell University（康奈尔大学高级计算中心）；Kavli Institute at Cornell for Nanoscale Science, Cornell University（康奈尔大学卡夫利纳米科学研究所）
Scientific Computing and Imaging Institute（科学计算与成像研究所）；University of Utah（犹他大学）
The manifold assumption for high-dimensional data assumes that the data is generated by varying a set of parameters obtained from a low-dimensional latent space. Deep generative models (DGMs) are widely used to learn data representations in an unsupervised way. DGMs parameterize the underlying low-dimensional manifold in the data space using bottleneck architectures such as variational autoencoders (VAEs). The bottleneck dimension for VAEs is treated as a hyperparameter that depends on the dataset and is fixed at design time after extensive tuning. As the intrinsic dimensionality of most real-…
