In this paper, we propose the physics informed adversarial training (PIAT) of neural networks for solving nonlinear differential equations (NDE). It is well-known that the standard training of neural networks results in non-smooth functions. Adversarial training (AT) is an established defense mechanism against adversarial attacks, which could also help in making the solution smooth. AT include augmenting the training mini-batch with a perturbation that makes the network output mismatch the desired output adversarially. Unlike formal AT, which relies only on the training data, here we encode the governing physical laws in the form of nonlinear differential equations using automatic differentiation in the adversarial network architecture. We compare PIAT with PINN to indicate the effectiveness of our method in solving NDEs for up to 10 dimensions. Moreover, we propose weight decay and Gaussian smoothing to demonstrate the PIAT advantages. The code repository is available at https://github.com/rohban-lab/PIAT.
Universidade Federal do Rio de Janeiro（里约热内卢联邦大学）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
With the increasing amount of available data and advances in computing capabilities, deep neural networks (DNNs) have been successfully employed to solve challenging tasks in various areas, including healthcare, climate, and finance. Nevertheless, state-of-the-art DNNs are susceptible to quasi-imperceptible perturbed versions of the original images -- adversarial examples. These perturbations of the network input can lead to disastrous implications in critical areas where wrong decisions can directly affect human lives. Adversarial training is the most efficient solution to defend the network …
Dalian University of Technology（大连理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；JD Explore Academy（京东探索研究院）
Physics-informed neural networks (PINNs) provide a deep learning framework for numerically solving partial differential equations (PDEs), and have been widely used in a variety of PDE problems. However, there still remain some challenges in the application of PINNs: 1) the mechanism of PINNs is unsuitable (at least cannot be directly applied) to exploiting a small size of (usually very few) extra informative samples to refine the networks; and 2) the efficiency of training PINNs often becomes low for some complicated PDEs. In this paper, we propose the generative adversarial physics-informed n…
Columbia University（哥伦比亚大学）；Data Science Institute, Columbia University（哥伦比亚大学数据科学研究所）
This paper proposes the TrafficFlowGAN, a physics-informed flow based generative adversarial network (GAN), for uncertainty quantification (UQ) of dynamical systems. TrafficFlowGAN adopts a normalizing flow model as the generator to explicitly estimate the data likelihood. This flow model is trained to maximize the data likelihood and to generate synthetic data that can fool a convolutional discriminator. We further regularize this training process using prior physics information, so-called physics-informed deep learning (PIDL). To the best of our knowledge, we are the first to propose an inte…
In the context of aircraft system performance assessment, deep learning technologies allow to quickly infer models from experimental measurements, with less detailed system knowledge than usually required by physics-based modeling. However, this inexpensive model development also comes with new challenges regarding model trustworthiness. This work presents a novel approach, physics-guided adversarial machine learning (ML), that improves the confidence over the physics consistency of the model. The approach performs, first, a physics-guided adversarial testing phase to search for test inputs re…
School of Statistics and Data Science, LPMC, KLMDASR and LEBPS, Nankai University（南开大学统计与数据科学学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
When and Why Adversarial Training Improves PINNs: A Neural Tangent Kernel Perspective
School of Mathematics and Statistics, Beijing Institute of Technology, China（北京理工大学数学与统计学院，中国）；School of Professional Education and Executive Development The Hong Kong Polytechnic University, China（香港理工大学专业教育学院及管理发展学院，中国）；Department of Computer Science & UCL AI Centre, University College London, UK（伦敦大学学院计算机科学系及UCL人工智能中心，英国）
There has been a recent surge in adversarial attacks on deep learning based automatic speech recognition (ASR) systems. These attacks pose new challenges to deep learning security and have raised significant concerns in deploying ASR systems in safety-critical applications. In this work, we introduce WaveGuard: a framework for detecting adversarial inputs that are crafted to attack ASR systems. Our framework incorporates audio transformation functions and analyses the ASR transcriptions of the original and transformed audio to detect adversarial inputs. We demonstrate that our defense framewor…
This paper extends the methodology to use physics-informed enhanced super-resolution generative adversarial networks (PIESRGANs) for LES subfilter modeling in turbulent flows with finite-rate chemistry and shows a successful application to a non-premixed temporal jet case. This is an important topic considering the need for more efficient and carbon-neutral energy devices to fight the climate change. Multiple a priori and a posteriori results are presented and discussed. As part of this, the impact of the underlying mesh on the prediction quality is emphasized, and a multi-mesh approach is dev…
Learning Without Adversarial Training: A Physics-Informed Neural Network for Secure Power System State Estimation under False Data Injection Attacks
KIOS Center of Excellence, University of Cyprus（塞浦路斯大学KIOS研究中心）；CEMSE, KAUST（卡塔尔大学能源科学与工程学院）；KIOS Center of Excellence, ECE Dept., University of Cyprus（塞浦路斯大学KIOS研究中心电子与计算机工程系）
Georgia Institute of Technology（佐治亚理工学院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Georgia（佐治亚大学）；Symantec CAML Group（赛门铁克 CAML 集团）
Graph modeling allows numerous security problems to be tackled in a general way, however, little work has been done to understand their ability to withstand adversarial attacks. We design and evaluate two novel graph attacks against a state-of-the-art network-level, graph-based detection system. Our work highlights areas in adversarial machine learning that have not yet been addressed, specifically: graph-based clustering techniques, and a global feature space where realistic attackers without perfect knowledge must be accounted for (by the defenders) in order to be practical. Even though less…
In this paper, we study a physics-informed algorithm for Wasserstein Generative Adversarial Networks (WGANs) for uncertainty quantification in solutions of partial differential equations. By using groupsort activation functions in adversarial network discriminators, network generators are utilized to learn the uncertainty in solutions of partial differential equations observed from the initial/boundary data. Under mild assumptions, we show that the generalization error of the computed generator converges to the approximation error of the network with high probability, when the number of sample…
We present a novel variant of Domain Adversarial Networks with impactful improvements to the loss functions, training paradigm, and hyperparameter optimization. New loss functions are defined for both forks of the DANN network, the label predictor and domain classifier, in order to facilitate more rapid gradient descent, provide more seamless integration into modern neural networking frameworks, and allow previously unavailable inferences into network behavior. Using these loss functions, it is possible to extend the concept of 'domain' to include arbitrary user defined labels applicable to su…
Neural networks' lack of robustness against attacks raises concerns in security-sensitive settings such as autonomous vehicles. While many countermeasures may look promising, only a few withstand rigorous evaluation. Defenses using random transformations (RT) have shown impressive results, particularly BaRT (Raff et al., 2019) on ImageNet. However, this type of defense has not been rigorously evaluated, leaving its robustness properties poorly understood. Their stochastic properties make evaluation more challenging and render many proposed attacks on deterministic models inapplicable. First, w…
We present a deep learning framework for quantifying and propagating uncertainty in systems governed by non-linear differential equations using physics-informed neural networks. Specifically, we employ latent variable models to construct probabilistic representations for the system states, and put forth an adversarial inference procedure for training them on data, while constraining their predictions to satisfy given physical laws expressed by partial differential equations. Such physics-informed constraints provide a regularization mechanism for effectively training deep generative models as …
Aalborg University（奥尔堡大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Implementations of SGD on distributed systems create new vulnerabilities, which can be identified and misused by one or more adversarial agents. Recently, it has been shown that well-known Byzantine-resilient gradient aggregation schemes are indeed vulnerable to informed attackers that can tailor the attacks (Fang et al., 2020; Xie et al., 2020b). We introduce MixTailor, a scheme based on randomization of the aggregation strategies that makes it impossible for the attacker to be fully informed. Deterministic schemes can be integrated into MixTailor on the fly without introducing any additional…
StablePDENet: Enhancing Neural Operator Stability through Physics-Informed Residual-Sensitivity Regularization
Department of Mathematics, The Hong Kong University of Science and Technology（香港科技大学数学系）；Algorithms of Machine Learning and Autonomous Driving Research Lab（机器学习与自动驾驶算法研究实验室）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳-香港协同创新研究院）
Stanford University（斯坦福大学）；Center for Turbulence Research（湍流研究中心）
Generative Adversarial Networks (GANs) have been widely used for generating photo-realistic images. A variant of GANs called super-resolution GAN (SRGAN) has already been used successfully for image super-resolution where low resolution images can be upsampled to a $4\times$ larger image that is perceptually more realistic. However, when such generative models are used for data describing physical processes, there are additional known constraints that models must satisfy including governing equations and boundary conditions. In general, these constraints may not be obeyed by the generated data…
A Residual Guided strategy with Generative Adversarial Networks in training Physics-Informed Transformer Networks
College of Science, China Agricultural University（中国农业大学理学院）；College of Information and Electrical Engineering, China Agricultural University（中国农业大学信息与电气工程学院）；College of Water Resources and Civil Engineering, China Agricultural University（中国农业大学水利与土木工程学院）
Water Resources Research Center（水资源研究中心）；University of Hawai’i at Mānoa（夏威夷大学马诺阿分校）；Yeungnam University（岭南大学）；Sandia National Laboratories（桑迪亚国家实验室）
Stochastic network modeling is often limited by high computational costs to generate a large number of networks enough for meaningful statistical evaluation. In this study, Deep Convolutional Generative Adversarial Networks (DCGANs) were applied to quickly reproduce drainage networks from the already generated network samples without repetitive long modeling of the stochastic network model, Gibb's model. In particular, we developed a novel connectivity-informed method that converts the drainage network images to the directional information of flow on each node of the drainage network, and then…
Harbin Institute of Technology（哈尔滨工业大学）；Peking University（北京大学）
Seismic full waveform inversion (FWI) is a powerful geophysical imaging technique that produces high-resolution subsurface models by iteratively minimizing the misfit between the simulated and observed seismograms. Unfortunately, conventional FWI with least-squares function suffers from many drawbacks such as the local-minima problem and computation of explicit gradient. It is particularly challenging with the contaminated measurements or poor starting models. Recent works relying on partial differential equations and neural networks show promising performance for two-dimensional FWI. Inspired…
Osaka University（大阪大学）；The Institute of Scientific and Industrial Research, Osaka University（大阪大学科学与工业研究所）
Recently, researchers have utilized neural networks to accurately solve partial differential equations (PDEs), enabling the mesh-free method for scientific computation. Unfortunately, the network performance drops when encountering a high nonlinearity domain. To improve the generalizability, we introduce the novel approach of employing multi-task learning techniques, the uncertainty-weighting loss and the gradients surgery, in the context of learning PDE solutions. The multi-task scheme exploits the benefits of learning shared representations, controlled by cross-stitch modules, between multip…
Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH（于利希研究中心于利希超级计算中心）；RWTH Aachen University（亚琛工业大学）
Models for finite-rate-chemistry in underresolved flows still pose one of the main challenges for predictive simulations of complex configurations. The problem gets even more challenging if turbulence is involved. This work advances the recently developed PIESRGAN modeling approach to turbulent premixed combustion. For that, the physical information processed by the network and considered in the loss function are adjusted, the training process is smoothed, and especially effects from density changes are considered. The resulting model provides good results for a priori and a posteriori tests o…
AdRo-FL: Informed and Secure Client Selection for Federated Learning in the Presence of Adversarial Aggregator
College of Computing and Mathematics, King Fahd University of Petroleum & Minerals（计算机与数学学院，国王法赫德石油与矿物大学）；School of Electronic Engineering and Computer Science, Queen Mary University of London（电子工程与计算机科学学院，女王玛丽大学）；Department of Computer Science, Carnegie Mellon University（计算机科学系，卡内基梅隆大学）
Uncertainty Quantification In Surface Landmines and UXO Classification Using MC Dropout
Rochester Institute of Technology, Chester F. Carlson Center for Imaging Science（罗切斯特理工学院，切斯特·F·卡森成像科学中心）；Jadavpur University, Department of Computer Science（贾梵布尔大学，计算机科学系）
Adversarial Disentanglement by Backpropagation with Physics-Informed Variational Autoencoder
Department of Civil Engineering and Geosciences（土木工程与地球科学系）；Technical University of Delft（代尔夫特理工大学）；Department of Engineering（工程系）；University of Cambridge（剑桥大学）
Turbulence is still one of the main challenges for accurately predicting reactive flows. Therefore, the development of new turbulence closures which can be applied to combustion problems is essential. Data-driven modeling has become very popular in many fields over the last years as large, often extensively labeled, datasets became available and training of large neural networks became possible on GPUs speeding up the learning process tremendously. However, the successful application of deep neural networks in fluid dynamics, for example for subgrid modeling in the context of large-eddy simula…
The accurate prediction of small scales in underresolved flows is still one of the main challenges in predictive simulations of complex configurations. Over the last few years, data-driven modeling has become popular in many fields as large, often extensively labeled datasets are now available and training of large neural networks has become possible on graphics processing units (GPUs) that speed up the learning process tremendously. In fact, the successful application of deep neural networks in fluid dynamics, such as for underresolved reactive flows, is still challenging. This work advances …
NASA Langley Research Center（美国国家航空航天局兰利研究中心）；University of Puerto Rico (Mayagüez)（波多黎各大学（马亚圭斯））
While standard generative adversarial networks (GANs) rely solely on training data to learn unknown probability distributions, physics-informed GANs (PI-GANs) encode physical laws in the form of stochastic partial differential equations (PDEs) using auto differentiation. By relating observed data to unobserved quantities of interest through PDEs, PI-GANs allow for the estimation of underlying probability distributions without their direct measurement (i.e. inverse problems). The scalable nature of GANs allows high-dimensional, spatially-dependent probability distributions (i.e., random fields)…
Recent approaches in text-to-speech (TTS) synthesis employ neural network strategies to vocode perceptually-informed spectrogram representations directly into listenable waveforms. Such vocoding procedures create a computational bottleneck in modern TTS pipelines. We propose an alternative approach which utilizes generative adversarial networks (GANs) to learn mappings from perceptually-informed spectrograms to simple magnitude spectrograms which can be heuristically vocoded. Through a user study, we show that our approach significantly outperforms naïve vocoding strategies while being hundred…
RAMS: Residual-based adversarial-gradient moving sample method for scientific machine learning in solving partial differential equations
Department of Civil and Environmental Engineering, Hong Kong Polytechnic University（香港理工大学土木与环境工程系）；Department of Statistics and Data Science, Yale University（耶鲁大学统计与数据科学系）
Thermodynamics-informed super-resolution of scarce temporal dynamics data
ESI Group（ESI集团）；Aragon Institute of Engineering Research (I3A)（阿拉贡工程研究所）；Universidad de Zaragoza（萨拉戈萨大学）；Universidad Politécnica de Madrid（马德里理工大学）；ENSAM Institute of Technology（法国国立高等机械与航空技术学院）
This paper aims to quantify uncertainty in traffic state estimation (TSE) using the generative adversarial network based physics-informed deep learning (PIDL). The uncertainty of the focus arises from fundamental diagrams, in other words, the mapping from traffic density to velocity. To quantify uncertainty for the TSE problem is to characterize the robustness of predicted traffic states. Since its inception, generative adversarial networks (GAN) have become a popular probabilistic machine learning framework. In this paper, we will inform the GAN based predictions using stochastic traffic flow…
Brown University（布朗大学）；Massachusetts Institute of Technology（麻省理工学院）；Pacific Northwest National Laboratory（西北太平洋国家实验室）
We propose a new method for inferring the governing stochastic ordinary differential equations (SODEs) by observing particle ensembles at discrete and sparse time instants, i.e., multiple "snapshots". Particle coordinates at a single time instant, possibly noisy or truncated, are recorded in each snapshot but are unpaired across the snapshots. By training a physics-informed generative model that generates "fake" sample paths, we aim to fit the observed particle ensemble distributions with a curve in the probability measure space, which is induced from the inferred particle dynamics. We employ …
Graz University of Technology（格拉茨工业大学）；Know Center GmbH（知识中心有限公司）；RWTH Aachen（亚琛工业大学）；GESIS - Leibniz Institute for the Social Sciences（GESIS-莱布尼茨社会科学研究所）
Recent work has shown that graph neural networks (GNNs) are vulnerable to adversarial attacks on graph data. Common attack approaches are typically informed, i.e. they have access to information about node attributes such as labels and feature vectors. In this work, we study adversarial attacks that are uninformed, where an attacker only has access to the graph structure, but no information about node attributes. Here the attacker aims to exploit structural knowledge and assumptions, which GNN models make about graph data. In particular, literature has shown that structural node centrality and…
University of Maryland（马里兰大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Considerable research has been devoted to deep learning-based predictive models for system prognostics and health management in the reliability and safety community. However, there is limited study on the utilization of deep learning for system reliability assessment. This paper aims to bridge this gap and explore this new interface between deep learning and system reliability assessment by exploiting the recent advances of physics-informed deep learning. Particularly, we present an approach to frame system reliability assessment in the context of physics-informed deep learning and discuss the…
Solutions to differential equations are of significant scientific and engineering relevance. Physics-Informed Neural Networks (PINNs) have emerged as a promising method for solving differential equations, but they lack a theoretical justification for the use of any particular loss function. This work presents Differential Equation GAN (DEQGAN), a novel method for solving differential equations using generative adversarial networks to "learn the loss function" for optimizing the neural network. Presenting results on a suite of twelve ordinary and partial differential equations, including the no…
