There is growing interest in data-driven weather prediction (DDWP), for example using convolutional neural networks such as U-NETs that are trained on data from models or reanalysis. Here, we propose 3 components to integrate with commonly used DDWP models in order to improve their physical consistency and forecast accuracy. These components are 1) a deep spatial transformer added to the latent space of the U-NETs to preserve a property called equivariance, which is related to correctly capturing rotations and scalings of features in spatio-temporal data, 2) a data-assimilation (DA) algorithm to ingest noisy observations and improve the initial conditions for next forecasts, and 3) a multi-time-step algorithm, which combines forecasts from DDWP models with different time steps through DA, improving the accuracy of forecasts at short intervals. To show the benefit/feasibility of each component, we use geopotential height at 500~hPa (Z500) from ERA5 reanalysis and examine the short-term forecast accuracy of specific setups of the DDWP framework. Results show that the equivariance-preserving networks (U-STNs) clearly outperform the U-NETs, for example improving the forecast skill by $45\%$. Using a sigma-point ensemble Kalman (SPEnKF) algorithm for DA and U-STN as the forward model, we show that stable, accurate DA cycles are achieved even with high observation noise. The DDWP+DA framework substantially benefits from large ($O(1000)$) ensembles that are inexpensively generated with the data-driven forward model in each DA cycle. The multi-time-step DDWP+DA framework also shows promises, e.g., it reduces the average error by factors of 2-3.
Huracan: A skillful end-to-end data-driven system for ensemble data assimilation and weather prediction
Microsoft Corporation（微软公司）；Department of Earth System Science, Ministry of Education Key Laboratory for Earth System Modeling, Institute for Global Change Studies（教育部地球系统科学部门地球系统建模重点实验室，全球变化研究机构）；Tsinghua University（清华大学）
FuXiWeather2: Learning accurate atmospheric state estimation for operational global weather forecasting
Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）；Artificial Intelligence Innovation and Incubation Institute（人工智能创新与孵化院）；Fudan University（复旦大学）；FuXi Intelligent Computing Technology Co., Ltd.（FuXi智能计算技术有限公司）
Virginia Tech（弗吉尼亚理工大学）；Oden Institute for Computational Engineering and Sciences, University of Texas at Austin（德克萨斯大学奥斯汀分校奥登计算工程与科学研究所）
Data assimilation is the process of fusing information from imperfect computer simulations with noisy, sparse measurements of reality to obtain improved estimates of the state or parameters of a dynamical system of interest. The data assimilation procedures used in many geoscience applications, such as numerical weather forecasting, are variants of the our-dimensional variational (4D-Var) algorithm. The cost of solving the underlying 4D-Var optimization problem is dominated by the cost of repeated forward and adjoint model runs. This motivates substituting the evaluations of the physical model…
Generative assimilation and prediction for weather and climate
Chinese Academy of Sciences（中国科学院）；Nanjing University（南京大学）；Technical University of Munich（慕尼黑工业大学）；Potsdam Institute for Climate Impact Research（波茨坦气候影响研究所）；Chinese Academy of Meteorological Sciences（中国气象科学研究院）；National University of Defense Technology（国防科技大学）；Peking University（北京大学）；National Climate Center（国家气候中心）；Xiong’an Institute of Meteorological Artificial Intelligence（雄安气象人工智能研究院）
Learning more physically realistic dynamics in machine-learning based weather forecasting with latent-space constraints
Department of Earth and Environmental Engineering, Columbia University（哥伦比亚大学地球与环境工程系）；Learning the Earth with Artificial Intelligence and Physics (LEAP) Center, Columbia University（人工智能与物理联合地球学习中心（LEAP）, 哥伦比亚大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
SolarSeer: Ultrafast and accurate 24-hour solar irradiance forecasts outperforming numerical weather prediction across the USA
Microsoft（微软公司）；Tsinghua University（清华大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；University of California, Berkeley（加州大学伯克利分校）；Taiyuan University of Technology（太原科技大学）
RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges
The Hong Kong University of Science and Technology（香港科学与技术大学）；Nanjing University（南京大学）；Southern University of Science and Technology（南方科技大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai TechWind Technology Co., Ltd.（上海技风科技有限公司）
DAWP: A framework for global observation forecasting via Data Assimilation and Weather Prediction in satellite observation space
Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong Shanghai AI Laboratory（香港中文大学上海人工智能实验室）；Shanghai AI Laboratory（上海人工智能实验室）
PEAR: Equal Area Weather Forecasting on the Sphere
VERSES AI；Department of Mathematical Sciences（数学科学系）；Chalmers University of Technology（楚姆勒斯技术大学）；University of Gothenburg（哥德堡大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Department of Physics and Astronomy（物理与天文学系）；Recohere
Meta-Learning Fourier Neural Operators for Hessian Inversion and Enhanced Variational Data Assimilation
School of Computational Science and Engineering, McMaster University（计算科学与工程学院，麦斯特大学）；Mathematics and Statistics, McMaster University（数学与统计学，麦斯特大学）；Department of Chemistry and Chemical Biology, McMaster University（化学与生物化学系，麦斯特大学）
本文提出元学习 FNO 框架近似跨 DA 问题的逆 Hessian，以初始化 CG；线性平流实验显示 FNO-CG 显著降低误差与迭代数，尤其适用于病态场景。
Uncertainty-Aware End-to-End AI Weather Forecasting: Disentangling Observation and Model Contributions
Fraunhofer Heinrich-Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））
Virginia Polytechnic Institute and State University（弗吉尼亚理工学院暨州立大学）；Argonne National Laboratory（阿贡国家实验室）；Statistical and Applied Mathematical Science Institute（统计与应用数学科学研究所）
Data assimilation plays a key role in large-scale atmospheric weather forecasting, where the state of the physical system is estimated from model outputs and observations, and is then used as initial condition to produce accurate future forecasts. The Ensemble Kalman Filter (EnKF) provides a practical implementation of the statistical solution of the data assimilation problem and has gained wide popularity as. This success can be attributed to its simple formulation and ease of implementation. EnKF is a Monte-Carlo algorithm that solves the data assimilation problem by sampling the probability…
FlowDAS: A Stochastic Interpolant-based Framework for Data Assimilation
University of Michigan（密歇根大学）；Department of Electrical Engineering and Computer Science（电气工程与计算机科学系）；Peking University（北京大学）；Department of Physics（物理系）；National Biomedical Imaging Center（国家生物医学成像中心）
Fuel moisture has a major influence on the behavior of wildland fires and is an important underlying factor in fire risk assessment. We propose a method to assimilate dead fuel moisture content observations from remote automated weather stations (RAWS) into a time-lag fuel moisture model. RAWS are spatially sparse and a mechanism is needed to estimate fuel moisture content at locations potentially distant from observational stations. This is arranged using a trend surface model (TSM), which allows us to account for the effects of topography and atmospheric state on the spatial variability of f…
Google Research（谷歌研究院）；Technical University of Munich（慕尼黑工业大学）；Harvard University（哈佛大学）
Variational data assimilation optimizes for an initial state of a dynamical system such that its evolution fits observational data. The physical model can subsequently be evolved into the future to make predictions. This principle is a cornerstone of large scale forecasting applications such as numerical weather prediction. As such, it is implemented in current operational systems of weather forecasting agencies across the globe. However, finding a good initial state poses a difficult optimization problem in part due to the non-invertible relationship between physical states and their correspo…
In-situ ocean wave observations are critical to improve model skill and validate remote sensing wave measurements. Historically, such observations are extremely sparse due to the large costs and complexity of traditional wave buoys and sensors. In this work, we present a recently deployed network of free-drifting satellite-connected surface weather buoys that provide long-dwell coverage of surface weather in the northern Pacific Ocean basin. To evaluate the leading-order improvements to model forecast skill using this distributed sensor network, we implement a widely-used data assimilation tec…
The review describes the development of ideas Gury Ivanovich Marchuk in the field of variational data assimilation for ocean models applied in particular in coupled models for long-range weather forecasts. Particular attention is paid to the optimization of boundary conditions on rigid boundaries. As idealized and realistic model configurations are considered. It is shown that the optimization allows us to determine the most sensitive model operators and bring the model solution closer to the assimilated data.
Ensemble data assimilation methods such as the Ensemble Kalman Filter (EnKF) are a key component of probabilistic weather forecasting. They represent the uncertainty in the initial conditions by an ensemble which incorporates information coming from the physical model with the latest observations. High-resolution numerical weather prediction models ran at operational centers are able to resolve non-linear and non-Gaussian physical phenomena such as convection. There is therefore a growing need to develop ensemble assimilation algorithms able to deal with non-Gaussianity while staying computati…
Three-Dimensional Variational Data Assimilation with Rapid Update Cycling for Short-Range Precipitation Forecasting: A Case Study of Heavy Rainfall in Bali, Indonesia
This paper addresses the impact of assimilating data from the Earth Networks Total Lightning Network (ENTLN) during two cases of severe weather. Data from the ENTLN serve as a substitute for those from the upcoming launch of the GOES Lightning Mapper (GLM). We use the Weather Research and Forecast (WRF) model and variational data assimilation techniques at 9 km spatial resolution. The main goal is to examine the potential impact of lightning observations from the future GLM. Previous efforts to assimilate lightning observations mainly utilized nudging approaches. We develop three more sophisti…
The space weather community has begun to use frontier methods such as data assimilation, machine learning, and ensemble modeling to advance current operational forecasting efforts. This was highlighted by a multi-disciplinary session at the 2017 American Geophysical Union Meeting, 'Frontier Solar-Terrestrial Science Enabled by the Combination of Data-Driven Techniques and Physics-Based Understanding', with considerable discussion surrounding ensemble techniques. Here ensemble methods are described in detail; using a set of predictions to improve on a single-model output, for example taking a s…
Improving Typhoon Predictions by Integrating Data-Driven Machine Learning Models with Physics Models Based on the Spectral Nudging and Data Assimilation
Online machine-learning forecast uncertainty estimation for sequential data assimilation
Servicio Meteorológico Nacional（国家气象局）；Universidad de Buenos Aires（布宜诺斯艾利斯大学）；Instituto Franco-Argentino para el Estudio del Clima y sus Impactos (IRL 3351 IFAECI)（法阿气候及其影响研究所（IRL 3351 IFAECI））；Universidad Nacional del Nordeste（东北国立大学）；Centro de Investigaciones del Mar y la Atmósfera（海洋与大气研究中心）；IMT Atlantique（大西洋国立高等工业技术学院）；Inria（法国国家信息与自动化研究所）
