Constructing accurate and generalizable approximators for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalization. We demonstrate the performance of these methods in terms of generalization accuracy by solving the 2D time-dependent Brusselator reaction-diffusion system with uncertainty sources, modeling an autocatalytic chemical reaction between two species. We first propose an extension of the m-PCE by constructing a mapping between latent spaces formed by two separate embeddings of input functions and output QoIs. To enhance the accuracy of the DeepONet, we introduce weight self-adaptivity in the loss function. We demonstrate that the performance of m-PCE and DeepONet is comparable for cases of relatively smooth input-output mappings. However, when highly non-smooth dynamics is considered, DeepONet shows higher accuracy. We also find that for m-PCE, modest over-parameterization leads to better generalization, both within and outside of distribution, whereas aggressive over-parameterization leads to over-fitting. In contrast, an even highly over-parameterized DeepONet leads to better generalization for both smooth and non-smooth dynamics. Furthermore, we compare the performance of the above models with another operator learning model, the Fourier Neural Operator, and show that its over-parameterization also leads to better generalization. Our studies show that m-PCE can provide very good accuracy at very low training cost, whereas a highly over-parameterized DeepONet can provide better accuracy and robustness to noise but at higher training cost. In both methods, the inference cost is negligible.
Microsoft Autonomous Systems and Robotics Research（微软自主系统与机器人研究部）；Microsoft Research AI4Science（微软研究院AI4Science）
Partial differential equations (PDEs) are central to describing complex physical system simulations. Their expensive solution techniques have led to an increased interest in deep neural network based surrogates. However, the practical utility of training such surrogates is contingent on their ability to model complex multi-scale spatio-temporal phenomena. Various neural network architectures have been proposed to target such phenomena, most notably Fourier Neural Operators (FNOs), which give a natural handle over local & global spatial information via parameterization of different Fourier mode…
Assessing Emulator Design and Training for Modal Aerosol Microphysics Parameterizations in E3SMv2
Advanced Computing, Mathematics, and Data Division（先进计算、数学与数据部门）；Pacific Northwest National Laboratory（太平洋西北国家实验室）；Earth and Biological Sciences Directorate（地球与生命科学 Directorate）；Lawrence Berkeley National Laboratory（伯克利国家实验室）
Department of Fluid Dynamics（流体动力学系）；Center for Frontier AI（前沿人工智能中心）；Department of Mechanical Engineering（机械工程系）；University of Colorado Denver（科罗拉多大学丹佛分校）；Department of Engineering Mechanics（工程力学系）
Computational Intelligence (CI) techniques have shown great potential as a surrogate model of expensive physics simulation, with demonstrated ability to make fast predictions, albeit at the expense of accuracy in some cases. For many scientific and engineering problems involving geometrical design, it is desirable for the surrogate models to precisely describe the change in geometry and predict the consequences. In that context, we develop graph neural networks (GNNs) as fast surrogate models for physics simulation, which allow us to directly train the models on 2/3D geometry designs that are …
Source Distribution Estimation by Posterior Averaging
University of Bern（伯尔尼大学）；Inselspital, Bern University Hospital（伯尔尼大学附属 Inselspital 医院）；Graduate School for Cellular and Biomedical Sciences (GCB)（细胞与生物医学科学研究生院）；Diabetes Center Berne（伯尔尼糖尿病中心）
A deep-learning-based surrogate model is developed and applied for predicting dynamic subsurface flow in channelized geological models. The surrogate model is based on deep convolutional and recurrent neural network architectures, specifically a residual U-Net and a convolutional long short term memory recurrent network. Training samples entail global pressure and saturation maps, at a series of time steps, generated by simulating oil-water flow in many (1500 in our case) realizations of a 2D channelized system. After training, the `recurrent R-U-Net' surrogate model is shown to be capable of …
Hierarchical Bayesian networks and neural networks with stochastic hidden units are commonly perceived as two separate types of models. We show that either of these types of models can often be transformed into an instance of the other, by switching between centered and differentiable non-centered parameterizations of the latent variables. The choice of parameterization greatly influences the efficiency of gradient-based posterior inference; we show that they are often complementary to eachother, we clarify when each parameterization is preferred and show how inference can be made robust. In t…
The Ohio State University（俄亥俄州立大学）；Argonne National Laboratory（阿贡国家实验室）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
We propose GNN-Surrogate, a graph neural network-based surrogate model to explore the parameter space of ocean climate simulations. Parameter space exploration is important for domain scientists to understand the influence of input parameters (e.g., wind stress) on the simulation output (e.g., temperature). The exploration requires scientists to exhaust the complicated parameter space by running a batch of computationally expensive simulations. Our approach improves the efficiency of parameter space exploration with a surrogate model that predicts the simulation outputs accurately and efficien…
FIGNN: Feature-Specific Interpretability for Graph Neural Network Surrogate Models
Department of Mechanical Engineering, Pennsylvania State University, University Park, 16802, PA, USA（宾夕法尼亚州立大学机械工程系）；College of Information Sciences and Technology, Pennsylvania State University, University Park, 16802, PA, USA（宾夕法尼亚州立大学信息科学与技术学院）；Power Systems Division, Argonne National Laboratory, Lemont, 60439, IL, USA（阿贡国家实验室电力系统分部）
The use of deep neural network (DNN) models as surrogates for linear and nonlinear structural dynamical systems is explored. The goal is to develop DNN based surrogates to predict structural response, i.e., displacements and accelerations, for given input (harmonic) excitations. In particular, the focus is on the development of efficient network architectures using fully-connected, sparsely-connected, and convolutional network layers, and on the corresponding training strategies that can provide a balance between the overall network complexity and prediction accuracy in the target dataspaces. …
Influence functions approximate the effect of training samples in test-time predictions and have a wide variety of applications in machine learning interpretability and uncertainty estimation. A commonly-used (first-order) influence function can be implemented efficiently as a post-hoc method requiring access only to the gradients and Hessian of the model. For linear models, influence functions are well-defined due to the convexity of the underlying loss function and are generally accurate even across difficult settings where model changes are fairly large such as estimating group influences. …
In complex large-scale systems such as climate, important effects are caused by a combination of confounding processes that are not fully observable. The identification of sources from observations of system state is vital for attribution and prediction, which inform critical policy decisions. The difficulty of these types of inverse problems lies in the inability to isolate sources and the cost of simulating computational models. Surrogate models may enable the many-query algorithms required for source identification, but data challenges arise from high dimensionality of the state and source,…
Variational Neural Belief Parameterizations for Robust Dexterous Grasping under Multimodal Uncertainty
Department of Electrical & Computer Engineering and Institute for Systems Research, University of Maryland, College Park, MD, USA（电气与计算机工程系和系统研究所，马里兰大学，College Park, MD, USA）；Maryland Applied Graduate Engineering, A. James Clark School of Engineering, University of Maryland, College Park, MD, USA（马里兰应用研究生工程学院，A. James Clark工程学院，马里兰大学，College Park, MD, USA）
