Neural operators (NOs) excel at learning mappings between function spaces, serving as efficient forward solution approximators for PDE-governed systems. However, as black-box solvers, they offer limited insight into the underlying physical mechanism, due to the lack of interpretable representations of the physical parameters that drive the system. To tackle this challenge, we propose a new paradigm for learning disentangled representations from NO parameters, thereby effectively solving an inverse problem. Specifically, we introduce DisentangO, a novel hyper-neural operator architecture designed to unveil and disentangle latent physical factors of variation embedded within the black-box neural operator parameters. At the core of DisentangO is a multi-task NO architecture that distills the varying parameters of the governing PDE through a task-wise adaptive layer, alongside a variational autoencoder that disentangles these variations into identifiable latent factors. By learning these disentangled representations, DisentangO not only enhances physical interpretability but also enables more robust generalization across diverse systems. Empirical evaluations across supervised, semi-supervised, and unsupervised learning contexts show that DisentangO effectively extracts meaningful and interpretable latent features, bridging the gap between predictive performance and physical understanding in neural operator frameworks.
Rochester Institute of Technology（罗切斯特理工学院）；Dalhousie University（达尔豪斯大学）
To improve the ability of VAE to disentangle in the latent space, existing works mostly focus on enforcing independence among the learned latent factors. However, the ability of these models to disentangle often decreases as the complexity of the generative factors increases. In this paper, we investigate the little-explored effect of the modeling capacity of a posterior density on the disentangling ability of the VAE. We note that the independence within and the complexity of the latent density are two different properties we constrain when regularizing the posterior density: while the former…
Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning
Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Guangdong University of Technology（广东工业大学）
Department of Computer Science and Engineering, University of Texas at Arlington, United States（德克萨斯理工大学计算机科学与工程系，美国）；School of Computing, University of Georgia, United States（佐治亚大学计算机学院，美国）
We propose a novel VAE-based deep auto-encoder model that can learn disentangled latent representations in a fully unsupervised manner, endowed with the ability to identify all meaningful sources of variation and their cardinality. Our model, dubbed Relevance-Factor-VAE, leverages the total correlation (TC) in the latent space to achieve the disentanglement goal, but also addresses the key issue of existing approaches which cannot distinguish between meaningful and nuisance factors of latent variation, often the source of considerable degradation in disentanglement performance. We tackle this …
University of Cambridge（剑桥大学）；Disney Research（迪士尼研究院）
We present a VAE architecture for encoding and generating high dimensional sequential data, such as video or audio. Our deep generative model learns a latent representation of the data which is split into a static and dynamic part, allowing us to approximately disentangle latent time-dependent features (dynamics) from features which are preserved over time (content). This architecture gives us partial control over generating content and dynamics by conditioning on either one of these sets of features. In our experiments on artificially generated cartoon video clips and voice recordings, we sho…
Multi-modal generative models represent an important family of deep models, whose goal is to facilitate representation learning on data with multiple views or modalities. However, current deep multi-modal models focus on the inference of shared representations, while neglecting the important private aspects of data within individual modalities. In this paper, we introduce a disentangled multi-modal variational autoencoder (DMVAE) that utilizes disentangled VAE strategy to separate the private and shared latent spaces of multiple modalities. We specifically consider the instance where the laten…
We focus on explicitly learning disentangled representation for natural image generation, where the underlying spatial structure and the rendering on the structure can be independently controlled respectively, yet using no tuple supervision. The setting is significant since tuple supervision is costly and sometimes even unavailable. However, the task is highly unconstrained and thus ill-posed. To address this problem, we propose to introduce an auxiliary domain which shares a common underlying-structure space with the target domain, and we make a partially shared latent space assumption. The k…
University of Oxford（牛津大学）；Alan Turing Institute（艾伦·图灵研究所）；University of Cambridge（剑桥大学）；Northeastern University（东北大学）；Stanford University（斯坦福大学）；Microsoft Research（微软研究院）；Deepmind
Variational autoencoders (VAEs) learn representations of data by jointly training a probabilistic encoder and decoder network. Typically these models encode all features of the data into a single variable. Here we are interested in learning disentangled representations that encode distinct aspects of the data into separate variables. We propose to learn such representations using model architectures that generalise from standard VAEs, employing a general graphical model structure in the encoder and decoder. This allows us to train partially-specified models that make relatively strong assumpti…
Department of Mathematics（数学系）；Hunan University（湖南大学）；Academy for Multidisciplinary Studies（跨学科研究学院）；Beijing National Center for Applied Mathematics（北京应用数学中心）；Capital Normal University（首都师范大学）
Shearlet Neural Operators for Anisotropic-Shock-Dominated and Multi-scale parametric partial differential equations
Coordination of Mathematical and Computational Methods National Laboratory of Scientific Computing（数学与计算方法协调国家科学计算实验室）；Systems and Computational Engineering Program, COPPE/UFRJ, Federal University of Rio de Janeiro（系统与计算工程系，COPPE/UFRJ，里约热内卢联邦大学）
Samsung AI Center Cambridge（三星剑桥人工智能中心）；Rutgers University（罗格斯大学）
We deal with the problem of learning the underlying disentangled latent factors that are shared between the paired bi-modal data in cross-modal retrieval. Our assumption is that the data in both modalities are complex, structured, and high dimensional (e.g., image and text), for which the conventional deep auto-encoding latent variable models such as the Variational Autoencoder (VAE) often suffer from difficulty of accurate decoder training or realistic synthesis. A suboptimally trained decoder can potentially harm the model's capability of identifying the true factors. In this paper we propos…
The classical development of neural networks has primarily focused on learning mappings between finite-dimensional Euclidean spaces. Recently, this has been generalized to neural operators that learn mappings between function spaces. For partial differential equations (PDEs), neural operators directly learn the mapping from any functional parametric dependence to the solution. Thus, they learn an entire family of PDEs, in contrast to classical methods which solve one instance of the equation. In this work, we formulate a new neural operator by parameterizing the integral kernel directly in Fou…
Adaptation of uncertainty-penalized Bayesian information criterion for parametric partial differential equation discovery
Graduate School of Information Science and Technology, Osaka University（大阪大学信息科学与技术研究生院）；SANKEN (The Institute of Scientific and Industrial Research), Osaka University（大阪大学产业科学研究所（SANKEN））
Implementing deep neural networks for learning the solution maps of parametric partial differential equations (PDEs) turns out to be more efficient than using many conventional numerical methods. However, limited theoretical analyses have been conducted on this approach. In this study, we investigate the expressive power of deep rectified quadratic unit (ReQU) neural networks for approximating the solution maps of parametric PDEs. The proposed approach is motivated by the recent important work of G. Kutyniok, P. Petersen, M. Raslan and R. Schneider (Gitta Kutyniok, Philipp Petersen, Mones Rasl…
University of Vienna（维也纳大学）；University of Oxford（牛津大学）；Austrian Academy of Sciences（奥地利科学院）
We present a deep learning algorithm for the numerical solution of parametric families of high-dimensional linear Kolmogorov partial differential equations (PDEs). Our method is based on reformulating the numerical approximation of a whole family of Kolmogorov PDEs as a single statistical learning problem using the Feynman-Kac formula. Successful numerical experiments are presented, which empirically confirm the functionality and efficiency of our proposed algorithm in the case of heat equations and Black-Scholes option pricing models parametrized by affine-linear coefficient functions. We sho…
Peking University（北京大学）；Southern University of Science and Technology（南方科技大学）；Peng Cheng Laboratory（鹏城实验室）
Data-driven methods have recently made great progress in the discovery of partial differential equations (PDEs) from spatial-temporal data. However, several challenges remain to be solved, including sparse noisy data, incomplete candidate library, and spatially- or temporally-varying coefficients. In this work, a new framework, which combines neural network, genetic algorithm and adaptive methods, is put forward to address all of these challenges simultaneously. In the framework, a trained neural network is utilized to calculate derivatives and generate a large amount of meta-data, which solve…
Indian Institute of Technology Delhi（德里印度理工学院）；Yardi School of Artificial Intelligence (ScAI)（亚迪人工智能学院（ScAI））
With massive advancements in sensor technologies and Internet-of-things, we now have access to terabytes of historical data; however, there is a lack of clarity in how to best exploit the data to predict future events. One possible alternative in this context is to utilize operator learning algorithm that directly learn nonlinear mapping between two functional spaces; this facilitates real-time prediction of naturally arising complex evolutionary dynamics. In this work, we introduce a novel operator learning algorithm referred to as the Wavelet Neural Operator (WNO) that blends integral kernel…
We propose a framework to analyze how multivariate representations disentangle ground-truth generative factors. A quantitative analysis of disentanglement has been based on metrics designed to compare how one variable explains each generative factor. Current metrics, however, may fail to detect entanglement that involves more than two variables, e.g., representations that duplicate and rotate generative factors in high dimensional spaces. In this work, we establish a framework to analyze information sharing in a multivariate representation with Partial Information Decomposition and propose a n…
Technische Universität Berlin（柏林工业大学）；University of Tromsø（特罗姆瑟大学）；University of Vienna（维也纳大学）
We derive upper bounds on the complexity of ReLU neural networks approximating the solution maps of parametric partial differential equations. In particular, without any knowledge of its concrete shape, we use the inherent low-dimensionality of the solution manifold to obtain approximation rates which are significantly superior to those provided by classical neural network approximation results. Concretely, we use the existence of a small reduced basis to construct, for a large variety of parametric partial differential equations, neural networks that yield approximations of the parametric sol…
One of the main challenges in using deep learning-based methods for simulating physical systems and solving partial differential equations (PDEs) is formulating physics-based data in the desired structure for neural networks. Graph neural networks (GNNs) have gained popularity in this area since graphs offer a natural way of modeling particle interactions and provide a clear way of discretizing the continuum models. However, the graphs constructed for approximating such tasks usually ignore long-range interactions due to unfavorable scaling of the computational complexity with respect to the n…
Peking University（北京大学）；Huawei Technologies Co. Ltd（华为技术有限公司）；Beijing International Center for Mathematical Research, Peking University（北京国际数学研究中心（北京大学））；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）
Many important problems in science and engineering require solving the so-called parametric partial differential equations (PDEs), i.e., PDEs with different physical parameters, boundary conditions, shapes of computation domains, etc. Recently, building learning-based numerical solvers for parametric PDEs has become an emerging new field. One category of methods such as the Deep Galerkin Method (DGM) and Physics-Informed Neural Networks (PINNs) aim to approximate the solution of the PDEs. They are typically unsupervised and mesh-free, but require going through the time-consuming network traini…
Technische Universität Berlin（柏林工业大学）；University of Vienna（维也纳大学）；University of Tromsø（特罗姆瑟大学）
We perform a comprehensive numerical study of the effect of approximation-theoretical results for neural networks on practical learning problems in the context of numerical analysis. As the underlying model, we study the machine-learning-based solution of parametric partial differential equations. Here, approximation theory predicts that the performance of the model should depend only very mildly on the dimension of the parameter space and is determined by the intrinsic dimension of the solution manifold of the parametric partial differential equation. We use various methods to establish compa…
On the training of physics-informed neural operators for solving parametric partial differential equations
College of Civil Engineering, Tongji University（同济大学土木工程学院）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Institute for Foundations of Data Science, Yale University（耶鲁大学数据科学基础研究所）
Deep operator networks (DeepONets) are receiving increased attention thanks to their demonstrated capability to approximate nonlinear operators between infinite-dimensional Banach spaces. However, despite their remarkable early promise, they typically require large training data-sets consisting of paired input-output observations which may be expensive to obtain, while their predictions may not be consistent with the underlying physical principles that generated the observed data. In this work, we propose a novel model class coined as physics-informed DeepONets, which introduces an effective r…
Physics-Guided Discovery of Highly Nonlinear Parametric Partial Differential Equations
Carnegie Mellon University（卡内基梅隆大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；Hefei University of Technology（合肥工业大学）；RealAI（瑞莱智慧）；Tsinghua University（清华大学）
Cluster Attention Neural Operators for Solving Parametric Partial Differential Equations
University of Chinese Academy of Sciences（中国科学院大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；International School for Advanced Studies (SISSA)（国际高等研究学院（SISSA））；Zhongyuan University of Technology（中原工学院）
Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent Parametric Partial Differential Equations
University of Stuttgart（斯图加特大学）；Stuttgart Center for Simulation Science (SimTech)（斯图加特模拟科学中心）；International Max Planck Research School for Intelligent Systems (IMPRS-IS)（国际马克斯·普朗克智能系统研究院）
