We propose a hierarchical normalizing flow model for generating molecular graphs. The model produces new molecular structures from a single-node graph by recursively splitting every node into two. All operations are invertible and can be used as plug-and-play modules. The hierarchical nature of the latent codes allows for precise changes in the resulting graph: perturbations in the top layer cause global structural changes, while perturbations in the consequent layers change the resulting molecule marginally. The proposed model outperforms existing generative graph models on the distribution learning task. We also show successful experiments on global and constrained optimization of chemical properties using latent codes of the model.
Fractal Flow: Hierarchical and Interpretable Normalizing Flow via Topic Modeling and Recursive Strategy
School of Mathematics Harbin Institute of Technology（哈尔滨工业大学数学学院）；School of Mathematics and Institute for Artificial Intelligence Harbin Instiute of Technology（哈尔滨工业大学数学学院和人工智能研究院）；School of Earth and Space Sciences and Institute for Artificial Intelligence Peking University（北京大学地球和空间科学学院和人工智能研究院）
University of Amsterdam（阿姆斯特丹大学）；Flatiron Institute（熨斗研究所）；Princeton University（普林斯顿大学）
We propose parameterizing the population distribution of the gravitational wave population modeling framework (Hierarchical Bayesian Analysis) with a normalizing flow. We first demonstrate the merit of this method on illustrative experiments and then analyze four parameters of the latest LIGO/Virgo data release: primary mass, secondary mass, redshift, and effective spin. Our results show that despite the small and notoriously noisy dataset, the posterior predictive distributions (assuming a prior over the parameters of the flow) of the observed gravitational wave population recover structure t…
McGill University（麦吉尔大学）；Mila（米拉研究所）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
The choice of approximate posterior distributions plays a central role in stochastic variational inference (SVI). One effective solution is the use of normalizing flows \cut{defined on Euclidean spaces} to construct flexible posterior distributions. However, one key limitation of existing normalizing flows is that they are restricted to the Euclidean space and are ill-equipped to model data with an underlying hierarchical structure. To address this fundamental limitation, we present the first extension of normalizing flows to hyperbolic spaces. We first elevate normalizing flows to hyperbolic …
Conditional Normalizing Flows for Active Learning of Coarse-Grained Molecular Representations
Institute of Nanotechnology, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院纳米技术研究所）；Institute of Theoretical Informatics, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院理论信息学研究所）
Hierarchical Gaussian Mixture Normalizing Flow Modeling for Unified Anomaly Detection
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；School of Intelligent Manufacturing, Wuxi Vocational College of Science and Technology（无锡科技职业学院智能制造学院）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院、教育部人工智能重点实验室）
Based on the manifold hypothesis, real-world data often lie on a low-dimensional manifold, while normalizing flows as a likelihood-based generative model are incapable of finding this manifold due to their structural constraints. So, one interesting question arises: $\textit{"Can we find sub-manifold(s) of data in normalizing flows and estimate the density of the data on the sub-manifold(s)?"}$. In this paper, we introduce two approaches, namely per-pixel penalized log-likelihood and hierarchical training, to answer the mentioned question. We propose a single-step method for joint manifold lea…
Survival analysis, or time-to-event modelling, is a classical statistical problem that has garnered a lot of interest for its practical use in epidemiology, demographics or actuarial sciences. Recent advances on the subject from the point of view of machine learning have been concerned with precise per-individual predictions instead of population studies, driven by the rise of individualized medicine. We introduce here a conditional normalizing flow based estimate of the time-to-event density as a way to model highly flexible and individualized conditional survival distributions. We use a nove…
Tencent（腾讯）；National University of Singapore（新加坡国立大学）；MIT（麻省理工学院）
Deep Gaussian processes (DGPs), a hierarchical composition of GP models, have successfully boosted the expressive power of their single-layer counterpart. However, it is impossible to perform exact inference in DGPs, which has motivated the recent development of variational inference-based methods. Unfortunately, either these methods yield a biased posterior belief or it is difficult to evaluate their convergence. This paper introduces a new approach for specifying flexible, arbitrarily complex, and scalable approximate posterior distributions. The posterior distribution is constructed through…
PRHLT Research Center（PRHLT研究中心）；Universitat Politècnica de València（瓦伦西亚理工大学）；University of Warwick（华威大学）；The Alan Turing Institute（阿兰·图灵研究所）
Gaussian Processes (GPs) can be used as flexible, non-parametric function priors. Inspired by the growing body of work on Normalizing Flows, we enlarge this class of priors through a parametric invertible transformation that can be made input-dependent. Doing so also allows us to encode interpretable prior knowledge (e.g., boundedness constraints). We derive a variational approximation to the resulting Bayesian inference problem, which is as fast as stochastic variational GP regression (Hensman et al., 2013; Dezfouli and Bonilla,2015). This makes the model a computationally efficient alternati…
Computing excited states of molecules using normalizing flows
Center for Free-Electron Laser Science CFEL, Deutsches Elektronen-Synchrotron DESY（自由电子激光科学中心 CFEL，德意志电子同步辐射研究中心）；Department of Chemistry, Universität Hamburg（汉堡大学化学系）；Center for Ultrafast Imaging, Universität Hamburg（超快成像中心，汉堡大学）；Department of Physics, Universität Hamburg（汉堡大学物理系）；Department of Mathematics, Universität Hamburg（汉堡大学数学系）
Argonne National Laboratory（阿贡国家实验室）；Brookhaven National Laboratory（布鲁克海文国家实验室）；University of Connecticut（康涅狄格大学）
We propose using Normalizing Flows as a trainable kernel within the molecular dynamics update of Hamiltonian Monte Carlo (HMC). By learning (invertible) transformations that simplify our dynamics, we can outperform traditional methods at generating independent configurations. We show that, using a carefully constructed network architecture, our approach can be easily scaled to large lattice volumes with minimal retraining effort. The source code for our implementation is publicly available online at https://github.com/nftqcd/fthmc.
Amortized Sampling with Transferable Normalizing Flows
University of Oxford（牛津大学）；Université de Montréal（蒙特利尔大学）；Mila - Quebec AI Institute（魁北克AI研究所）；Freie Universität Berlin（柏林自由大学）；Valence Labs（Valence实验室）；AITHYRA；Institut Courtois（科特伊研究所）
Freie Universität Berlin（柏林自由大学）；Rice University（莱斯大学）
Normalizing flows are a promising tool for modeling probability distributions in physical systems. While state-of-the-art flows accurately approximate distributions and energies, applications in physics additionally require smooth energies to compute forces and higher-order derivatives. Furthermore, such densities are often defined on non-trivial topologies. A recent example are Boltzmann Generators for generating 3D-structures of peptides and small proteins. These generative models leverage the space of internal coordinates (dihedrals, angles, and bonds), which is a product of hypertori and c…
The sampling of probability distributions specified up to a normalization constant is an important problem in both machine learning and statistical mechanics. While classical stochastic sampling methods such as Markov Chain Monte Carlo (MCMC) or Langevin Dynamics (LD) can suffer from slow mixing times there is a growing interest in using normalizing flows in order to learn the transformation of a simple prior distribution to the given target distribution. Here we propose a generalized and combined approach to sample target densities: Stochastic Normalizing Flows (SNF) -- an arbitrary sequence …
Reaction coordinate flows for model reduction of molecular kinetics
Shanghai Jiao Tong University（上海交通大学）；Tongji University（同济大学）；Freie Universität Berlin（柏林自由大学）；Rice University（莱斯大学）；Microsoft Research AI4Science（微软研究院AI4Science）
We propose a framework using normalizing-flow based models, SELF-Referencing Embedded Strings, and multi-objective optimization that efficiently generates small molecules. With an initial training set of only 100 small molecules, FastFlows generates thousands of chemically valid molecules in seconds. Because of the efficient sampling, substructure filters can be applied as desired to eliminate compounds with unreasonable moieties. Using easily computable and learned metrics for druglikeness, synthetic accessibility, and synthetic complexity, we perform a multi-objective optimization to demonst…
Freie Universität Berlin（柏林自由大学）；Microsoft Research AI4Science（微软研究院AI4Science）
Coarse-grained (CG) molecular simulations have become a standard tool to study molecular processes on time- and length-scales inaccessible to all-atom simulations. Parameterizing CG force fields to match all-atom simulations has mainly relied on force-matching or relative entropy minimization, which require many samples from costly simulations with all-atom or CG resolutions, respectively. Here we present flow-matching, a new training method for CG force fields that combines the advantages of both methods by leveraging normalizing flows, a generative deep learning method. Flow-matching first t…
Normalizing flows, autoregressive models, variational autoencoders (VAEs), and deep energy-based models are among competing likelihood-based frameworks for deep generative learning. Among them, VAEs have the advantage of fast and tractable sampling and easy-to-access encoding networks. However, they are currently outperformed by other models such as normalizing flows and autoregressive models. While the majority of the research in VAEs is focused on the statistical challenges, we explore the orthogonal direction of carefully designing neural architectures for hierarchical VAEs. We propose Nouv…
Efficient Regression-Based Training of Normalizing Flows for Boltzmann Generators
Mila – Québec AI Institute（魁北克AI研究所）；Université de Montréal（蒙特利尔大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Oxford（牛津大学）；AITHYRA；HEC Montréal（蒙特利尔HEC）
National Yang Ming Chiao Tung University（阳明交通大学）；Poznań University of Technology（波兹南理工大学）
This paper introduces an end-to-end learned image compression system, termed ANFIC, based on Augmented Normalizing Flows (ANF). ANF is a new type of flow model, which stacks multiple variational autoencoders (VAE) for greater model expressiveness. The VAE-based image compression has gone mainstream, showing promising compression performance. Our work presents the first attempt to leverage VAE-based compression in a flow-based framework. ANFIC advances further compression efficiency by stacking and extending hierarchically multiple VAE's. The invertibility of ANF, together with our training str…
Generating a Boltzmann distribution in high dimension has recently been achieved with Normalizing Flows, which enable fast and exact computation of the generated density, and thus unbiased estimation of expectations. However, current implementations rely on accurate training data, which typically comes from computationally expensive simulations. There is therefore a clear incentive to train models with incomplete or no data by relying solely on the target density, which can be obtained from a physical energy model (up to a constant factor). For that purpose, we analyze the properties of standa…
Tokyo Institute of Technology（东京工业大学）；Preferred Networks, Inc.（Preferred Networks公司）
We propose GraphNVP, the first invertible, normalizing flow-based molecular graph generation model. We decompose the generation of a graph into two steps: generation of (i) an adjacency tensor and (ii) node attributes. This decomposition yields the exact likelihood maximization on graph-structured data, combined with two novel reversible flows. We empirically demonstrate that our model efficiently generates valid molecular graphs with almost no duplicated molecules. In addition, we observe that the learned latent space can be used to generate molecules with desired chemical properties.
We consider the problem of molecular graph generation using deep models. While graphs are discrete, most existing methods use continuous latent variables, resulting in inaccurate modeling of discrete graph structures. In this work, we propose GraphDF, a novel discrete latent variable model for molecular graph generation based on normalizing flow methods. GraphDF uses invertible modulo shift transforms to map discrete latent variables to graph nodes and edges. We show that the use of discrete latent variables reduces computational costs and eliminates the negative effect of dequantization. Comp…
Rigid Body Flows for Sampling Molecular Crystal Structures
Microsoft Research AI4Science（微软研究院AI4Science）；Freie Universität Berlin（柏林自由大学）；Qualcomm AI Research, an initiative from Qualcomm Technologies, Inc.（高通AI研究院（高通技术公司旗下研究机构））；University of Amsterdam（阿姆斯特丹大学）；Rice University（莱斯大学）
提出面向晶体中分子位置与取向的刚体归一化流，利用单位四元数流及其双重覆盖性质建模旋转，并在四面体系统和 TIP4P ice XI 的玻尔兹曼生成器中验证。
Institute of Nanotechnology, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院纳米技术研究所）；Institute of Theoretical Informatics, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院理论计算机科学研究所）
FragFM: Hierarchical Framework for Efficient Molecule Generation via Fragment-Level Discrete Flow Matching
Department of Chemistry, KAIST（韩国科学技术院化学系）；InnoCORE AI-CRED Institute, KAIST（韩国科学技术院InnoCORE AI-CRED研究所）；College of Pharmacy, Dongguk University（东国大学药学院）；Department of Data Science, KAIST（韩国科学技术院数据科学系）
Freie Universität Berlin（柏林自由大学）；Rice University（莱斯大学）
Normalizing flows are exact-likelihood generative neural networks which approximately transform samples from a simple prior distribution to samples of the probability distribution of interest. Recent work showed that such generative models can be utilized in statistical mechanics to sample equilibrium states of many-body systems in physics and chemistry. To scale and generalize these results, it is essential that the natural symmetries in the probability density -- in physics defined by the invariances of the target potential -- are built into the flow. We provide a theoretical sufficient crit…
Variational autoencoders (VAEs) have witnessed great success in performing the compression of image datasets. This success, made possible by the bits-back coding framework, has produced competitive compression performance across many benchmarks. However, despite this, VAE architectures are currently limited by a combination of coding practicalities and compression ratios. That is, not only do state-of-the-art methods, such as normalizing flows, often demonstrate out-performance, but the initial bits required in coding makes single and parallel image compression challenging. To remedy this, we …
In this paper, we propose Multiresolution Equivariant Graph Variational Autoencoders (MGVAE), the first hierarchical generative model to learn and generate graphs in a multiresolution and equivariant manner. At each resolution level, MGVAE employs higher order message passing to encode the graph while learning to partition it into mutually exclusive clusters and coarsening into a lower resolution that eventually creates a hierarchy of latent distributions. MGVAE then constructs a hierarchical generative model to variationally decode into a hierarchy of coarsened graphs. Importantly, our propos…
OnePlanet Research Center（OnePlanet研究中心）；imec-the Netherlands（荷兰微电子研究中心（imec荷兰分部））；Google Research（谷歌研究院）；Donders Institute for Brain, Cognition and Behaviour（唐德斯大脑、认知与行为研究所）；Radboud University（拉德堡德大学）
Normalizing flows have shown great success as general-purpose density estimators. However, many real world applications require the use of domain-specific knowledge, which normalizing flows cannot readily incorporate. We propose embedded-model flows (EMF), which alternate general-purpose transformations with structured layers that embed domain-specific inductive biases. These layers are automatically constructed by converting user-specified differentiable probabilistic models into equivalent bijective transformations. We also introduce gated structured layers, which allow bypassing the parts o…
Frequently, population studies feature pyramidally-organized data represented using Hierarchical Bayesian Models (HBM) enriched with plates. These models can become prohibitively large in settings such as neuroimaging, where a sample is composed of a functional MRI signal measured on 300 brain locations, across 4 measurement sessions, and 30 subjects, resulting in around 1 million latent parameters.Such high dimensionality hampers the usage of modern, expressive flow-based techniques.To infer parameter posterior distributions in this challenging class of problems, we designed a novel methodolo…
Transition Path Sampling with Boltzmann Generator-based MCMC Moves
School of Computation, Information and Technology, TUM（慕尼黑工业大学计算、信息与技术学院）；Computer Science and Artificial Intelligence Laboratory, MIT（麻省理工学院计算机科学与人工智能实验室）；Department of Computer Science, ETH Zürich（苏黎世联邦理工学院计算机科学系）
Inria（法国国家信息与自动化研究所）；CEA（法国原子能和替代能源委员会）；Université Paris-Saclay（巴黎萨克雷大学）；University of Liège（列日大学）
Inferring the parameters of a stochastic model based on experimental observations is central to the scientific method. A particularly challenging setting is when the model is strongly indeterminate, i.e. when distinct sets of parameters yield identical observations. This arises in many practical situations, such as when inferring the distance and power of a radio source (is the source close and weak or far and strong?) or when estimating the amplifier gain and underlying brain activity of an electrophysiological experiment. In this work, we present hierarchical neural posterior estimation (HNP…
We introduce the Hierarchically Interacting Particle Neural Network (HIP-NN) to model molecular properties from datasets of quantum calculations. Inspired by a many-body expansion, HIP-NN decomposes properties, such as energy, as a sum over hierarchical terms. These terms are generated from a neural network--a composition of many nonlinear transformations--acting on a representation of the molecule. HIP-NN achieves state-of-the-art performance on a dataset of 131k ground state organic molecules, and predicts energies with 0.26 kcal/mol mean absolute error. With minimal tuning, our model is als…
The problem of accelerating drug discovery relies heavily on automatic tools to optimize precursor molecules to afford them with better biochemical properties. Our work in this paper substantially extends prior state-of-the-art on graph-to-graph translation methods for molecular optimization. In particular, we realize coherent multi-resolution representations by interweaving the encoding of substructure components with the atom-level encoding of the original molecular graph. Moreover, our graph decoder is fully autoregressive, and interleaves each step of adding a new substructure with the pro…
Hierarchically combining semi-empirical methods and first-principles calculations we gain a novel and noteworthy picture of the molecular-level mechanisms that govern the water-gas-shift (WGS) and reverse water-gas-shift (r-WGS) reactions on Rh catalysts. Central to this picture is that the WGS and r-WGS follow two different dominant reaction mechanisms: WGS proceeds according to a carboxyl (COOH) mechanism, whereas r-WGS proceeds according to a redox (CO2 {\to} CO + O) mechanism. The obtained results furthermore underscore the danger of common first-principles analyses that focus on a priori …
We present CHAMPION: a software developed to automatically detect time-dependent bonds between atoms based on their dynamics, classify the local graph topology around them, and analyze the physicochemical properties of these topologies by statistical physics. In stark contrast to methodologies where bonds are detected based on static conditions such as cut-off distances, CHAMPION considers pairs of atoms to be bound only if they move together and act as a bound pair over time. Furthermore, the time-dependent global bond graph is possible to split into dynamically shifting connected components …
