U-Net Kalman Filter (UNetKF): An Example of Machine Learning-assisted Ensemble Data Assimilation
University Cooperation of Atmospheric Research（大学大气研究合作组织）；National Oceanic and Atmospheric Administration/Geophysical Fluid Dynamics Laboratory（美国国家海洋和大气管理局/地球物理流体动力学实验室）；Princeton University（普林斯顿大学）
Machine learning techniques have seen a tremendous rise in popularity in weather and climate sciences. Data assimilation (DA), which combines observations and numerical models, has great potential to incorporate machine learning and artificial intelligence (ML/AI) techniques. In this paper, we use U-Net, a type of convolutional neutral network (CNN), to predict the localized ensemble covariances for the Ensemble Kalman Filter (EnKF) algorithm. Using a 2-layer quasi-geostrophic model, U-Nets are trained using data from EnKF DA experiments. The trained U-Nets are then used to predict the flow-dependent localized error covariance matrices in U-Net Kalman Filter (UNetKF) experiments, which are compared to traditional 3-dimensional variational (3DVar), ensemble 3DVar (En3DVar) and EnKF methods. The performance of UNetKF can match or exceed that of 3DVar, En3DVar or EnKF. We also demonstrate that trained U-Nets can be transferred to a higher-resolution model for UNetKF implementation, which again performs competitively to 3DVar and EnKF, particularly for small ensemble sizes.
Data assimilation is concerned with sequentially estimating a temporally-evolving state. This task, which arises in a wide range of scientific and engineering applications, is particularly challenging when the state is high-dimensional and the state-space dynamics are unknown. This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics. In doing so, AD-EnKFs leverage the ability of ensemble …
University of Naples, Federico II, Complesso Universitario M. S. Angelo, via Cintia, 80126 Naples.
We focus on Partial Differential Equation (PDE) based Data Assimilatio problems (DA) solved by means of variational approaches and Kalman filter algorithm. Recently, we presented a Domain Decomposition framework (we call it DD-DA, for short) performing a decomposition of the whole physical domain along space and time directions, and joining the idea of Schwarz' methods and parallel in time approaches. For effective parallelization of DD-DA algorithms, the computational load assigned to subdomains must be equally distributed. Usually computational cost is proportional to the amount of data enti…
Online machine-learning forecast uncertainty estimation for sequential data assimilation
Servicio Meteorológico Nacional（国家气象局）；Universidad de Buenos Aires（布宜诺斯艾利斯大学）；Instituto Franco-Argentino para el Estudio del Clima y sus Impactos (IRL 3351 IFAECI)（法阿气候及其影响研究所（IRL 3351 IFAECI））；Universidad Nacional del Nordeste（东北国立大学）；Centro de Investigaciones del Mar y la Atmósfera（海洋与大气研究中心）；IMT Atlantique（大西洋国立高等工业技术学院）；Inria（法国国家信息与自动化研究所）
RWTH Aachen University（亚琛工业大学）；Technische Universität Braunschweig（布伦瑞克工业大学）；KAUST（阿卜杜拉国王科技大学）
This paper presents the machine learning-based ensemble conditional mean filter (ML-EnCMF) -- a filtering method based on the conditional mean filter (CMF) previously introduced in the literature. The updated mean of the CMF matches that of the posterior, obtained by applying Bayes' rule on the filter's forecast distribution. Moreover, we show that the CMF's updated covariance coincides with the expected conditional covariance. Implementing the EnCMF requires computing the conditional mean (CM). A likelihood-based estimator is prone to significant errors for small ensemble sizes, causing the f…
Virginia Tech（弗吉尼亚理工大学）；Computational Science Laboratory（计算科学实验室）
Data assimilation is a Bayesian inference process that obtains an enhanced understanding of a physical system of interest by fusing information from an inexact physics-based model, and from noisy sparse observations of reality. The multifidelity ensemble Kalman filter (MFEnKF) recently developed by the authors combines a full-order physical model and a hierarchy of reduced order surrogate models in order to increase the computational efficiency of data assimilation. The standard MFEnKF uses linear couplings between models, and is statistically optimal in case of Gaussian probability densities.…
Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1
Institute for Advanced Academic Research, Chiba University（千叶大学高等学术研究院）；Center for Environmental Remote Sensing, Chiba University（千叶大学环境遥感中心）；Research Institute of Disaster Medicine, Chiba University（千叶大学灾害医学研究所）；Graduate School of Science and Engineering, Chiba University（千叶大学科学与工程研究生院）
University of Colorado Denver（科罗拉多大学丹佛分校）；San José State University（圣何塞州立大学）；Colorado State University（科罗拉多州立大学）；University of Utah（犹他大学）；University of Southern California（南加州大学）
The current fuel moisture content (FMC) subsystems in WRF-SFIRE and its workflow system WRFx use a time-lag differential equation model with assimilation of data from FMC sensors on Remote Automated Weather Stations (RAWS) by the extended augmented Kalman filter. But the quality of the result is constrained by the limitations of the model and of the Kalman filter. We observe that the data flow in a system consisting of a model and the Kalman filter can be interpreted to be the same as the data flow in a recurrent neural network (RNN). Thus, instead of building more sophisticated models and dat…
Imperial College London（帝国理工学院）；University of Cambridge（剑桥大学）
There is an urgent need to build models to tackle Indoor Air Quality issue. Since the model should be accurate and fast, Reduced Order Modelling technique is used to reduce the dimensionality of the problem. The accuracy of the model, that represent a dynamic system, is improved integrating real data coming from sensors using Data Assimilation techniques. In this paper, we formulate a new methodology called Latent Assimilation that combines Data Assimilation and Machine Learning. We use a Convolutional neural network to reduce the dimensionality of the problem, a Long-Short-Term-Memory to buil…
Sorbonne University（索邦大学）；University of Reading（雷丁大学）；University of Utrecht（乌得勒支大学）；École des Ponts ParisTech（巴黎高科路桥学院）；EDF R&D（法国电力集团研发部）；Université Paris-Est（巴黎东大学）
A novel method, based on the combination of data assimilation and machine learning is introduced. The new hybrid approach is designed for a two-fold scope: (i) emulating hidden, possibly chaotic, dynamics and (ii) predicting their future states. The method consists in applying iteratively a data assimilation step, here an ensemble Kalman filter, and a neural network. Data assimilation is used to optimally combine a surrogate model with sparse noisy data. The output analysis is spatially complete and is used as a training set by the neural network to update the surrogate model. The two steps ar…
University of Cambridge（剑桥大学）；Technical University of Munich（慕尼黑工业大学）
Reduced-order models based on level-set methods are widely used tools to qualitatively capture and track the nonlinear dynamics of an interface. The aim of this paper is to develop a physics-informed, data-driven, statistically rigorous learning algorithm for state and parameter estimation with level-set methods. A Bayesian approach based on data assimilation is introduced. Data assimilation is enabled by the ensemble Kalman filter and smoother, which are used in their probabilistic formulations. The level-set data assimilation framework is verified in onedimensional and two-dimensional test c…
Nansen Environmental and Remote Sensing Center（南森环境与遥感中心）；Sorbonne Université（索邦大学）；Universitetet i Bergen（卑尔根大学）；Bjerknes Center for Climate Research（比耶克内斯气候研究中心）
Increasing the resolution of a model can improve the performance of a data assimilation system: first because model field are in better agreement with high resolution observations, then the corrections are better sustained and, with ensemble data assimilation, the forecast error covariances are improved. However, resolution increase is associated with a cubical increase of the computational costs. Here we are testing an approach inspired from images super-resolution techniques and called "Super-resolution data assimilation" (SRDA). Starting from a low-resolution forecast, a neural network (NN)…
Rice University（莱斯大学）；École Normale Supérieure（巴黎高等师范学院）；PSL University（巴黎文理研究大学）；California Institute of Technology（加州理工学院）
Data assimilation (DA) is a key component of many forecasting models in science and engineering. DA allows one to estimate better initial conditions using an imperfect dynamical model of the system and noisy/sparse observations available from the system. Ensemble Kalman filter (EnKF) is a DA algorithm that is widely used in applications involving high-dimensional nonlinear dynamical systems. However, EnKF requires evolving large ensembles of forecasts using the dynamical model of the system. This often becomes computationally intractable, especially when the number of states of the system is v…
School of Mathematics and Statistics University of Sydney（悉尼大学数学与统计学院）；Institute of Mathematics University of Potsdam（波茨坦大学数学研究所）
Data-driven prediction and physics-agnostic machine-learning methods have attracted increased interest in recent years achieving forecast horizons going well beyond those to be expected for chaotic dynamical systems. In a separate strand of research data-assimilation has been successfully used to optimally combine forecast models and their inherent uncertainty with incoming noisy observations. The key idea in our work here is to achieve increased forecast capabilities by judiciously combining machine-learning algorithms and data assimilation. We combine the physics-agnostic data-driven approac…
Flow Matching for Efficient and Scalable Data Assimilation
The Computing and Mathematical Sciences Department, California Institute of Technology（加州理工学院计算与数学科学系）；Department of Mathematics and Scientific Computing and Imaging Institute, University of Utah（犹他大学数学与科学计算系和成像研究所）
College of Information Science and Engineering, China University of Petroleum – Beijing（中国石油大学（北京）信息科学与工程学院）
In recent years, the prosperity of deep learning has revolutionized the Artificial Neural Networks. However, the dependence of gradients and the offline training mechanism in the learning algorithms prevents the ANN for further improvement. In this study, a gradient-free training framework based on data assimilation is proposed to avoid the calculation of gradients. In data assimilation algorithms, the error covariance between the forecasts and observations is used to optimize the parameters. Feedforward Neural Networks (FNNs) are trained by gradient decent, data assimilation algorithms (Ensem…
In recent years, machine learning methods represented by deep neural networks (DNN) have been a new paradigm of turbulence modeling. However, in the scenario of high Reynolds numbers, there are still some bottlenecks, including the lack of high-fidelity data and the convergence and stability problem in the coupling process of turbulence models and the RANS solvers. In this paper, we propose an improved ensemble kalman inversion method as a unified approach of data assimilation and turbulence modeling for separated flows at high Reynolds numbers. The trainable parameters of the DNN are optimize…
The reconstruction of the dynamics of an observed physical system as a surrogate model has been brought to the fore by recent advances in machine learning. To deal with partial and noisy observations in that endeavor, machine learning representations of the surrogate model can be used within a Bayesian data assimilation framework. However, these approaches require to consider long time series of observational data, meant to be assimilated all together. This paper investigates the possibility to learn both the dynamics and the state online, i.e. to update their estimates at any time, in particu…
The prediction of the gas production from mature gas wells, due to their complex end-of-life behavior, is challenging and crucial for operational decision making. In this paper, we apply a modified deep LSTM model for prediction of the gas flow rates in mature gas wells, including the uncertainties in input parameters. Additionally, due to changes in the system in time and in order to increase the accuracy and robustness of the prediction, the Ensemble Kalman Filter (EnKF) is used to update the flow rate predictions based on new observations. The developed approach was tested on the data from …
ANITI, Université de Toulouse（ANITI，图卢兹大学）；Atos BDS R&D AI4Sim（Atos BDS 研发 AI4Sim）；CERFACS（欧洲科学计算研究与高级培训中心）；UFTMIP；NVIDIA（英伟达）
Performing Data Assimilation (DA) at a low cost is of prime concern in Earth system modeling, particularly at the time of big data where huge quantities of observations are available. Capitalizing on the ability of Neural Networks techniques for approximating the solution of PDE's, we incorporate Deep Learning (DL) methods into a DA framework. More precisely, we exploit the latent structure provided by autoencoders (AEs) to design an Ensemble Transform Kalman Filter with model error (ETKF-Q) in the latent space. Model dynamics are also propagated within the latent space via a surrogate neural …
Small Ensemble-based Data Assimilation: A Machine Learning-Enhanced Data Assimilation Method with Limited Ensemble Size
Centre for Regional Oceans, Department of Ocean Science and Technology, and State Key Laboratory of Internet of Things for Smart City, University of Macau（澳门地区海洋研究中心、海洋科学与技术学院及智能城市物联网国家重点实验室，澳门大学）；School of Naval Architecture and Ocean Engineering, Huazhong University of Science and Technology（华中科技大学船舶与海洋工程学院）；Ningbo Institute of Dalian University of Technology（大连理工大学宁波研究院）；College of Civil Engineering, Zhejiang University of Technology（浙江工业大学土木工程学院）；Department of Naval Architecture and Ocean Engineering, Hongik University（成均馆大学船舶与海洋工程学院）；State Key Laboratory of Internet of Things for Smart City, University of Macau（智能城市物联网国家重点实验室，澳门大学）；Zhuhai UM Science and Technology Research Institute（珠海UM科技研究院）
Nodal Hydraulic Head Estimation through Unscented Kalman Filter for Data-driven Leak Localization in Water Networks
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息技术研究所）；University of Bucharest（布加勒斯特大学）；Institute for Logic and Data Science（逻辑与数据科学研究所）；Politehnica University of Bucharest（布加勒斯特理工大学）；Supervision, Safety and Automatic Control Research Center (CS2AC) - UPC（加泰罗尼亚理工大学监督、安全与自动控制研究中心）
