Ensemble data assimilation methods such as the Ensemble Kalman Filter (EnKF) are a key component of probabilistic weather forecasting. They represent the uncertainty in the initial conditions by an ensemble which incorporates information coming from the physical model with the latest observations. High-resolution numerical weather prediction models ran at operational centers are able to resolve non-linear and non-Gaussian physical phenomena such as convection. There is therefore a growing need to develop ensemble assimilation algorithms able to deal with non-Gaussianity while staying computationally feasible. In the present paper we address some of these needs by proposing a new hybrid algorithm based on the Ensemble Kalman Particle Filter. It is fully formulated in ensemble space and uses a deterministic scheme such that it has the ensemble transform Kalman filter (ETKF) instead of the stochastic EnKF as a limiting case. A new criterion for choosing the proportion of particle filter and ETKF update is also proposed. The new algorithm is implemented in the COSMO framework and numerical experiments in a quasi-operational convective-scale setup are conducted. The results show the feasibility of the new algorithm in practice and indicate a strong potential for such local hybrid methods, in particular for forecasting non-Gaussian variables such as wind and hourly precipitation.
Ensemble methods such as the Ensemble Kalman Filter (EnKF) are widely used for data assimilation in large-scale geophysical applications, as for example in numerical weather prediction (NWP). There is a growing interest for physical models with higher and higher resolution, which brings new challenges for data assimilation techniques because of the presence of non-linear and non-Gaussian features that are not adequately treated by the EnKF. We propose two new localized algorithms based on the Ensemble Kalman Particle Filter (EnKPF), a hybrid method combining the EnKF and the Particle Filter (P…
We study the ensemble Kalman filter (EnKF) algorithm for sequential data assimilation in a general situation, that is, for nonlinear forecast and measurement models with non-additive and non-Gaussian noises. Such applications traditionally force us to choose between inaccurate Gaussian assumptions that permit efficient algorithms (e.g., EnKF), or more accurate direct sampling methods which scale poorly with dimension (e.g., particle filters, or PF). We introduce a trimmed ensemble Kalman filter (TEnKF) which can interpolate between the limiting distributions of the EnKF and PF to facilitate ad…
Data assimilation is concerned with sequentially estimating a temporally-evolving state. This task, which arises in a wide range of scientific and engineering applications, is particularly challenging when the state is high-dimensional and the state-space dynamics are unknown. This paper introduces a machine learning framework for learning dynamical systems in data assimilation. Our auto-differentiable ensemble Kalman filters (AD-EnKFs) blend ensemble Kalman filters for state recovery with machine learning tools for learning the dynamics. In doing so, AD-EnKFs leverage the ability of ensemble …
Over the years data assimilation methods have been developed to obtain estimations of uncertain model parameters by taking into account a few observations of a model state. The most reliable methods of MCMC are computationally expensive. Sequential ensemble methods such as ensemble Kalman filers and particle filters provide a favourable alternative. However, Ensemble Kalman Filter has an assumption of Gaussianity. Ensemble Transform Particle Filter does not have this assumption and has proven to be highly beneficial for an initial condition estimation and a small number of parameter estimation…
The performance of ensemble-based data assimilation techniques that estimate the state of a dynamical system from partial observations depends crucially on the prescribed uncertainty of the model dynamics and of the observations. These are not usually known and have to be inferred. Many approaches have been proposed to tackle this problem, including fully Bayesian, likelihood maximization and innovation-based techniques. This work focuses on maximization of the likelihood function via the expectation-maximization (EM) algorithm to infer the model error covariance combined with ensemble Kalman …
We review the field of data assimilation (DA) from a Bayesian perspective and show that, in addition to its by now common application to state estimation, DA may be used for model selection. An important special case of the latter is the discrimination between a factual model --- which corresponds, to the best of the modeler's knowledge, to the situation in the actual world in which a sequence of events has occurred --- and a counterfactual model, in which a particular forcing or process might be absent or just quantitatively different from the actual world. Three different ensemble-DA methods…
State-space models can be used to incorporate subject knowledge on the underlying dynamics of a time series by the introduction of a latent Markov state-process. A user can specify the dynamics of this process together with how the state relates to partial and noisy observations that have been made. Inference and prediction then involves solving a challenging inverse problem: calculating the conditional distribution of quantities of interest given the observations. This article reviews Monte Carlo algorithms for solving this inverse problem, covering methods based on the particle filter and th…
We introduce a framework for Data Assimilation (DA) in which the data is split into multiple sets corresponding to low-rank projections of the state space. Algorithms are developed that assimilate some or all of the projected data, including an algorithm compatible with any generic DA method. The major application explored here is PROJ-PF, a projected Particle Filter. The PROJ-PF implementation assimilates highly informative but low-dimensional observations. The implementation considered here is based upon using projections corresponding to Assimilation in the Unstable Subspace (AUS). In the c…
Flow Matching for Efficient and Scalable Data Assimilation
The Computing and Mathematical Sciences Department, California Institute of Technology（加州理工学院计算与数学科学系）；Department of Mathematics and Scientific Computing and Imaging Institute, University of Utah（犹他大学数学与科学计算系和成像研究所）
We consider the problem of conditioning a geological process-based computer simulation, which produces basin models by simulating transport and deposition of sediments, to data. Emphasising uncertainty quantification, we frame this as a Bayesian inverse problem, and propose to characterize the posterior probability distribution of the geological quantities of interest by using a variant of the ensemble Kalman filter, an estimation method which linearly and sequentially conditions realisations of the system state to data. A test case involving synthetic data is used to assess the performance of…
A hybrid particle ensemble Kalman filter is developed for problems with medium non-Gaussianity, i.e. problems where the prior is very non-Gaussian but the posterior is approximately Gaussian. Such situations arise, e.g., when nonlinear dynamics produce a non-Gaussian forecast but a tight Gaussian likelihood leads to a nearly-Gaussian posterior. The hybrid filter starts by factoring the likelihood. First the particle filter assimilates the observations with one factor of the likelihood to produce an intermediate prior that is close to Gaussian, and then the ensemble Kalman filter completes the …
For particle filters and ensemble Kalman filters it is of practical importance to understand how and why data assimilation methods can be effective when used with a fixed small number of particles, since for many large-scale applications it is not practical to deploy algorithms close to the large particle limit asymptotic. In this paper we address this question for particle filters and, in particular, study their accuracy (in the small noise limit) and ergodicity (for noisy signal and observation) without appealing to the large particle number limit. We first overview the accuracy and minoriza…
Data assimilation combines information from models, measurements, and priors to estimate the state of a dynamical system such as the atmosphere. The Ensemble Kalman filter (EnKF) is a family of ensemble-based data assimilation approaches that has gained wide popularity due its simple formulation, ease of implementation, and good practical results. Most EnKF algorithms assume that the underlying probability distributions are Gaussian. Although this assumption is well accepted, it is too restrictive when applied to large nonlinear models, nonlinear observation operators, and large levels of unce…
We provide a clear and concise introduction to the subjects of inverse problems and data assimilation, and their inter-relations. The first part of our notes covers inverse problems; this refers to the study of how to estimate unknown model parameters from data. The second part of our notes covers data assimilation; this refers to a particular class of inverse problems in which the unknown parameter is the initial condition (and/or state) of a dynamical system, and the data comprises partial and noisy observations of the state. The third and final part of our notes describes the use of data as…
The Ensemble Kalman filter assumes the observations to be Gaussian random variables with a pre-specified mean and variance. In practice, observations may also have detection limits, for instance when a gauge has a minimum or maximum value. In such cases most data assimilation schemes discard out-of-range values, treating them as "not a number", at a loss of possibly useful qualitative information. The current work focuses on the development of a data assimilation scheme that tackles observations with a detection limit. We present the Ensemble Kalman Filter Semi-Qualitative (EnKF-SQ) and test i…
Parameter estimation has a high importance in the geosciences. The ensemble Kalman filter (EnKF) allows parameter estimation for large, time-dependent systems. For large systems, the EnKF is applied using small ensembles, which may lead to spurious correlations and, ultimately, to filter divergence. We present a thorough evaluation of the pilot point ensemble Kalman filter (PP-EnKF), a variant of the ensemble Kalman filter for parameter estimation. In this evaluation, we explicitly state the update equations of the PP-EnKF, discuss the differences of this update equation compared to the update…
Accurate runoff forecasting is crucial for reservoir operators as it allows optimized water management, flood control and hydropower generation. Land surface models in mountainous regions depend on climatic inputs such as precipitation, temperature and solar radiation to model the water and energy dynamics and produce runoff as output. With the rapid development of cheap electronics applied in various systems, such as Wireless Sensor Networks (WSNs), satellite and airborne technologies, the prospect of practically measuring spatial Snow Water Equivalent in a dense temporal scale is increasing.…
Data assimilation is a method that combines observations (that is, real world data) of a state of a system with model output for that system in order to improve the estimate of the state of the system and thereby the model output. The model is usually represented by a discretised partial differential equation. The data assimilation problem can be formulated as a large scale Bayesian inverse problem. Based on this interpretation we will derive the most important variational and sequential data assimilation approaches, in particular three-dimensional and four-dimensional variational data assimil…
For oceanographic applications, probabilistic forecasts typically have to deal with i) high-dimensional complex models, and ii) very sparse spatial observations. In search-and-rescue operations at sea, for instance, the short-term predictions of drift trajectories are essential to efficiently define search areas, but in-situ buoy observations provide only very sparse point measurements, while the mission is ongoing. Statistically optimal forecasts, including consistent uncertainty statements, rely on Bayesian methods for data assimilation to make the best out of both the complex mathematical m…
Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1
Institute for Advanced Academic Research, Chiba University（千叶大学高等学术研究院）；Center for Environmental Remote Sensing, Chiba University（千叶大学环境遥感中心）；Research Institute of Disaster Medicine, Chiba University（千叶大学灾害医学研究所）；Graduate School of Science and Engineering, Chiba University（千叶大学科学与工程研究生院）
A newly introduced stochastic data assimilation method, the Ensemble Kalman Filter Semi-Qualitative (EnKF-SQ) is applied to a realistic coupled ice-ocean model of the Arctic, the TOPAZ4 configuration, in a twin experiment framework. The method is shown to add value to range-limited thin ice thickness measurements, as obtained from passive microwave remote sensing, with respect to more trivial solutions like neglecting the out-of-range values or assimilating climatology instead. Some known properties inherent to the EnKF-SQ are evaluated: the tendency to draw the solution closer to the thicknes…
Fuel moisture is a major influence on the behavior of wildland fires and an important underlying factor in fire risk. We present a method to assimilate spatially sparse fuel moisture observations from remote automatic weather stations (RAWS) into the moisture model in WRF-SFIRE. WRF-SFIRE is a coupled atmospheric and fire behavior model which simulates the evolution of fuel moisture in idealized fuel species based on atmospheric state. The proposed method uses a modified trend surface model to estimate the fuel moisture field and its uncertainty based on currently available observations. At ea…
The ensemble Kalman filter (EnKF) is a data assimilation technique that uses an ensemble of models, updated with data, to track the time evolution of a usually non-linear system. It does so by using an empirical approximation to the well-known Kalman filter. However, its performance can suffer when the ensemble size is smaller than the state space, as is often necessary for computationally burdensome models. This scenario means that the empirical estimate of the state covariance is not full rank and possibly quite noisy. To solve this problem in this high dimensional regime, we propose a compu…
To represent the complex individual interactions in the dynamics of disease spread informed by data, the coupling of an epidemiological agent-based model with the ensemble Kalman filter is proposed. The statistical inference of the propagation of a disease by means of ensemble-based data assimilation systems has been studied in previous work. The models used are mostly compartmental models representing the mean field evolution through ordinary differential equations. These techniques allow to monitor the propagation of the infections from data and to estimate several parameters of epidemiologi…
In this paper we propose to numerically assess the performance of standard Gaussian approximations to probe the posterior distribution that arises from Bayesian data assimilation in petroleum reservoirs. In particular we assess the performance of (i) the linearization around the maximum a posterior estimate, (ii) the randomized maximum likelihood and (iii) standard ensemble Kalman filter-type methods. In order to fully resolve the posterior distribution we implement a state-of-the art MCMC method that scales well with respect to the dimension of the parameter space. Our implementation of the M…
