Numerical Weather Forecasting using Convolutional-LSTM with Attention and Context Matcher Mechanisms
Georgia Institute of Technology（佐治亚理工学院）；Bilkent University（比尔肯特大学）
Selim Furkan Tekin, Arda Fazla, Suleyman Serdar Kozat
备注：
- In our journal submission, we removed the integration of the observational data section since it was not used in the experiments. Thus, we also removed the authors from the paper who were responsible for that section. - In the second version, we also performed an experiment on WeatherBench. We compare our results with the Physical Weather Forecasting Models
采用高分辨率物理模型进行数值天气预报通常需要在超级计算机上消耗大量计算资源，这限制了其在大多数实际应用中的广泛使用。作为一种解决方案，深度学习方法的应用为该领域带来了创新性的解决思路。为此，我们提出了一种用于高分辨率时空天气数据预报的新型深度学习架构。我们的方法通过整合卷积长短期记忆网络（Convolutional Long-short Term Memory）与卷积神经网络，对传统的编码器-解码器结构进行了扩展。此外，我们还将注意力机制与上下文匹配器机制融入模型架构中。我们的Weather Model与ConvLSTM、TrajGRU和U-Net等基线深度学习模型相比，实现了显著的性能提升。我们的实验评估采用了大规模的真实世界基准数值天气数据集，即ERA5气压层小时数据集与WeatherBench。研究结果表明，该模型通过聚焦输入序列不同部分的注意力矩阵来建模大气环流，在识别时空相关性方面取得了实质性改进。我们还使用基准指标将该模型与高分辨率物理模型进行了对比，结果显示我们的Weather Model准确率高且易于解释。
英文摘要：
Numerical weather forecasting using high-resolution physical models often requires extensive computational resources on supercomputers, which diminishes their wide usage in most real-life applications. As a remedy, applying deep learning methods has revealed innovative solutions within this field. To this end, we introduce a novel deep learning architecture for forecasting high-resolution spatio-temporal weather data. Our approach extends the conventional encoder-decoder structure by integrating Convolutional Long-short Term Memory and Convolutional Neural Networks. In addition, we incorporate attention and context matcher mechanisms into the model architecture. Our Weather Model achieves significant performance improvements compared to baseline deep learning models, including ConvLSTM, TrajGRU, and U-Net. Our experimental evaluation involves high-scale, real-world benchmark numerical weather datasets, namely the ERA5 hourly dataset on pressure levels and WeatherBench. Our results demonstrate substantial improvements in identifying spatial and temporal correlations with attention matrices focusing on distinct parts of the input series to model atmospheric circulations. We also compare our model with high-resolution physical models using the benchmark metrics and show that our Weather Model is accurate and easy to interpret.
Long Short-Term Memory (LSTM) is a well-known method used widely on sequence learning and time series prediction. In this paper we deployed stacked LSTM model in an application of weather forecasting. We propose a 2-layer spatio-temporal stacked LSTM model which consists of independent LSTM models per location in the first LSTM layer. Subsequently, the input of the second LSTM layer is formed based on the combination of the hidden states of the first layer LSTM models. The experiments show that by utilizing the spatial information the prediction performance of the stacked LSTM model improves i…
Time Distributed Deep Learning Models for Purely Exogenous Forecasting: Application to Water Table Depth Prediction using Weather Image Time Series
Computer Science Department - University of Turin（都灵大学计算机科学系）；LISTIC Laboratory - Université Savoie Mont Blanc（萨瓦伊-蒙布莱大学LISTIC实验室）；Interuniversity Department of Regional and Urban Studies and Planning, Politecnico di Torino and University of Turin（都灵理工学院和都灵大学区域与城市研究与规划联合学院）
Agricultural Recommendation System based on Deep Learning: A Multivariate Weather Forecasting Approach
Chittagong University of Engineering & Technology（吉大港工程技术大学）；Khulna University of Engineering & Technology（库尔纳工程技术大学）；University of Illinois at Chicago（芝加哥大学伊利诺伊分校）；Khulna Agricultural University（库尔纳农业大学）；Edith Cowan University（伊迪丝·考恩大学）
Weather Prediction Using CNN-LSTM for Time Series Analysis: A Case Study on Delhi Temperature Data
Glasgow College, University of Electronic Science and Technology of China（电子科技大学格拉斯哥学院）；Viterbi School of Engineering, University of Southern California（南加州大学维特比工程学院）
Building an accurate load forecasting model with minimal underpredictions is vital to prevent any undesired power outages due to underproduction of electricity. However, the power consumption patterns of the residential sector contain fluctuations and anomalies making them challenging to predict. In this paper, we propose multiple Long Short-Term Memory (LSTM) frameworks with different asymmetric loss functions to impose a higher penalty on underpredictions. We also apply a density-based spatial clustering of applications with noise (DBSCAN) anomaly detection approach, prior to the load foreca…
Weather forecasting benefits us in various ways from farmers in cultivation and harvesting their crops to airlines to schedule their flights. Weather forecasting is a challenging task due to the chaotic nature of the atmosphere. Therefore lot of research attention has drawn to obtain the benefits and to overcome the challenges of weather forecasting. This paper compares ARIMA (Auto Regressive Integrated Moving Average) model and deep learning models to forecast temperature. The deep learning model consists of one dimensional convolutional layers to extract spatial features and LSTM layers to e…
Max-Planck-Institute for Biogeochemistry（马克斯·普朗克生物地球化学研究所）；ELLIS Unit Jena（耶拿欧洲学习与智能系统实验室）；Michael-Stifel-Center Jena for Data-driven and Simulation Science（耶拿米夏埃尔·施蒂费尔数据驱动与模拟科学中心）
Forecasting the state of vegetation in response to climate and weather events is a major challenge. Its implementation will prove crucial in predicting crop yield, forest damage, or more generally the impact on ecosystems services relevant for socio-economic functioning, which if absent can lead to humanitarian disasters. Vegetation status depends on weather and environmental conditions that modulate complex ecological processes taking place at several timescales. Interactions between vegetation and different environmental drivers express responses at instantaneous but also time-lagged effects…
University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；National Oceanic and Atmospheric Administration（美国国家海洋和大气管理局）
Poor air quality can have a significant impact on human health. The National Oceanic and Atmospheric Administration (NOAA) air quality forecasting guidance is challenged by the increasing presence of extreme air quality events due to extreme weather events such as wild fires and heatwaves. These extreme air quality events further affect human health. Traditional methods used to correct model bias make assumptions about linearity and the underlying distribution. Extreme air quality events tend to occur without a strong signal leading up to the event and this behavior tends to cause existing met…
The Pennsylvania State University（宾夕法尼亚州立大学）；National Center for Atmospheric Research（美国国家大气研究中心）
The rising temperature is one of the key indicators of a warming climate, and it can cause extensive stress to biological systems as well as built structures. Due to the heat island effect, it is most severe in urban environments compared to other landscapes due to the decrease in vegetation associated with a dense human-built environment. It is essential to adequately monitor the local temperature dynamics to mitigate risks associated with increasing temperatures, which can include short term strategy to protect people and animals, to long term strategy to how to build a new structure and cop…
Short-Term Regional Electricity Demand Forecasting in Argentina Using LSTM Networks
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Instituto de Investigaciones en Fisicoquímica de Córdoba (INFIQC), Universidad Nacional de Córdoba, Facultad de Ciencias Químicas, Departamento de Química Teórica y Computacional（阿根廷国家科学研究与技术理事会（CONICET）、科里亚物理化学研究所（INFIQC）、科里亚国立大学、化学科学学院、理论与计算化学系）
Currently the UK Electric market is guided by load (demand) forecasts published every thirty minutes by the regulator. A key factor in predicting demand is weather conditions, with forecasts published every hour. We present HYENA: a hybrid predictive model that combines feature engineering (selection of the candidate predictor features), mobile-window predictors and finally LSTM encoder-decoders to achieve higher accuracy with respect to mainstream models from the literature. HYENA decreased MAPE loss by 16\% and RMSE loss by 10\% over the best available benchmark model, thus establishing a ne…
Microsoft Corporation（微软公司）；Microsoft Turing（微软图灵）
We present the encoder-forecaster convolutional long short-term memory (LSTM) deep-learning model that powers Microsoft Weather's operational precipitation nowcasting product. This model takes as input a sequence of weather radar mosaics and deterministically predicts future radar reflectivity at lead times up to 6 hours. By stacking a large input receptive field along the feature dimension and conditioning the model's forecaster with predictions from the physics-based High Resolution Rapid Refresh (HRRR) model, we are able to outperform optical flow and HRRR baselines by 20-25% on multiple me…
This paper presents an engine able to forecast jointly the concentrations of the main pollutants harming people's health: nitrogen dioxyde (NO2), ozone (O3) and particulate matter (PM2.5 and PM10, which are respectively the particles whose diameters are below 2.5 um and 10 um respectively). The forecasts are performed on a regular grid (the results presented in the paper are produced with a 0.5° resolution grid over Europe and the United States) with a neural network whose architecture includes convolutional LSTM blocks. The engine is fed with the most recent air quality monitoring stations me…
This paper improves wind power prediction via weather forecast-contextualized Long Short-Term Memory Neural Network (LSTM) models. Initially, only wind power data was fed to a generic LSTM, but this model performed poorly, with erratic and naive behavior observed on even low-variance data sections. To address this issue, weather forecast data was added to better contextualize the power data, and LSTM modifications were made to address specific model shortcomings. These models were tested through both a Normalized Mean Absolute Error and the Naive Ratio (NR), which is a score introduced by this…
QLIF-CAST: Quantum Leaky-Integrate-and-Fire for Time-Series Weather Forecasting
eBrain Lab, Division of Engineering, New York University Abu Dhabi（eBrain实验室，工程系，纽约大学阿布扎克分校）；Center for Quantum and Topological Systems, NYUAD Research Institute, New York University Abu Dhabi（量子与拓扑系统中心，NYUAD研究学院，纽约大学阿布扎克分校）
Universidad Nacional de Colombia（哥伦比亚国立大学）；University of Florida（佛罗里达大学）
Accurate mechanisms for forecasting solar irradiance and insolation provide important information for the planning of renewable energy and agriculture projects as well as for environmental and socio-economical studies. This research introduces a pipeline for the one-day ahead forecasting of solar irradiance and insolation that only requires solar irradiance historical data for training. Furthermore, our approach is able to deal with missing data since it includes a data imputation state. In the prediction stage, we consider four data-driven approaches: Autoregressive Integrated Moving Average …
Model extraction attacks are a kind of attacks in which an adversary obtains a new model, whose performance is equivalent to that of a target model, via query access to the target model efficiently, i.e., fewer datasets and computational resources than those of the target model. Existing works have dealt with only simple deep neural networks (DNNs), e.g., only three layers, as targets of model extraction attacks, and hence are not aware of the effectiveness of recurrent neural networks (RNNs) in dealing with time-series data. In this work, we shed light on the threats of model extraction attac…
Maastricht University（马斯特里赫特大学）；Mathematics Centre Maastricht（马斯特里赫特数学中心）
Reliable weather forecasting is of great importance in science, business, and society. The best performing data-driven models for weather prediction tasks rely on recurrent or convolutional neural networks, where some of which incorporate attention mechanisms. In this work, we introduce a novel model based on Transformer architecture for weather forecasting. The proposed Tensorial Encoder Transformer (TENT) model is equipped with tensorial attention and thus it exploits the spatiotemporal structure of weather data by processing it in multidimensional tensorial format. We show that compared to …
Accurate air turbulence forecasting can help airlines avoid hazardous turbulence, guide the routes that keep passengers safe, maximize efficiency, and reduce costs. Traditional turbulence forecasting approaches heavily rely on painstakingly customized turbulence indexes, which are less effective in dynamic and complex weather conditions. The recent availability of high-resolution weather data and turbulence records allows more accurate forecasting of the turbulence in a data-driven way. However, it is a non-trivial task for developing a machine learning based turbulence forecasting system due …
National University of Sciences and Technology（国立科技大学）；Information Technology University (ITU)（信息技术大学（ITU））
Wheat is the main agricultural crop of Pakistan and is a staple food requirement of almost every Pakistani household making it the main strategic commodity of the country whose availability and affordability is the government's main priority. Wheat food availability can be vastly affected by multiple factors included but not limited to the production, consumption, financial crisis, inflation, or volatile market. The government ensures food security by particular policy and monitory arrangements, which keeps up purchase parity for the poor. Such arrangements can be made more effective if a dyna…
Deep Learning Meets Teleconnections: Improving S2S Predictions for European Winter Weather
TU Berlin（柏林工业大学）；ATB（莱布尼茨农业工程与生物经济研究所）；Leipzig University（莱比锡大学）；University of Reading（雷丁大学）；The Alan Turing Institute（阿兰·图灵研究所）；University of Potsdam（波茨坦大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）
CSIRO Data61（澳大利亚联邦科学与工业研究组织Data61）；The Australian National University（澳大利亚国立大学）；Defence Science and Technology Group, Department of Defence（国防部国防科学与技术集团）；CSIRO（澳大利亚联邦科学与工业研究组织）
Sea surface temperature (SST) variability plays a key role in the global weather and climate system, with phenomena such as El Niño-Southern Oscillation regarded as a major source of interannual climate variability at the global scale. The ability to be able to make long-range forecasts of sea surface temperature anomalies, especially those associated with extreme marine heatwave events, has potentially significant economic and societal benefits. We have developed a deep learning time series prediction model (Unet-LSTM) based on more than 70 years (1950-2021) of ECMWF ERA5 monthly mean sea sur…
Acadia Institute for Data Analytics（阿卡迪亚数据分析研究所）；Acadia University（阿卡迪亚大学）
Objective: To develop machine learning models that can predict the number of COVID-19 cases per day given the last 14 days of environmental and mobility data. Approach: COVID-19 data from four counties around Toronto, Ontario, were used. Data were prepared into daily records containing the number of new COVID case counts, patient demographic data, outdoor weather variables, indoor environment factors, and human movement based on cell mobility and public health restrictions. This data was analyzed to determine the most important variables and their interactions. Predictive models were developed…
Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）
The energy output a photo voltaic(PV) panel is a function of solar irradiation and weather parameters like temperature and wind speed etc. A general measure for solar irradiation called Global Horizontal Irradiance (GHI), customarily reported in Watt/meter$^2$, is a generic indicator for this intermittent energy resource. An accurate prediction of GHI is necessary for reliable grid integration of the renewable as well as for power market trading. While some machine learning techniques are well introduced along with the traditional time-series forecasting techniques, deep-learning techniques re…
Time series forecasting is widely used in the fields of equipment life cycle forecasting, weather forecasting, traffic flow forecasting, and other fields. Recently, some scholars have tried to apply Transformer to time series forecasting because of its powerful parallel training ability. However, the existing Transformer methods do not pay enough attention to the small time segments that play a decisive role in prediction, making it insensitive to small changes that affect the trend of time series, and it is difficult to effectively learn continuous time-dependent features. To solve this probl…
Time series modeling has entered an era of unprecedented growth in the size and complexity of data which require new modeling approaches. While many new general purpose machine learning approaches have emerged, they remain poorly understand and irreconcilable with more traditional statistical modeling approaches. We present a general class of exponential smoothed recurrent neural networks (RNNs) which are well suited to modeling non-stationary dynamical systems arising in industrial applications. In particular, we analyze their capacity to characterize the non-linear partial autocorrelation st…
Beyond Weather Correlation: A Comparative Study of Static and Temporal Neural Architectures for Fine-Grained Residential Energy Consumption Forecasting in Melbourne, Australia
School of Computing and Information Technology（计算与信息科技学院）
A deep learning model is applied for predicting block-level parking occupancy in real time. The model leverages Graph-Convolutional Neural Networks (GCNN) to extract the spatial relations of traffic flow in large-scale networks, and utilizes Recurrent Neural Networks (RNN) with Long-Short Term Memory (LSTM) to capture the temporal features. In addition, the model is capable of taking multiple heterogeneously structured traffic data sources as input, such as parking meter transactions, traffic speed, and weather conditions. The model performance is evaluated through a case study in Pittsburgh d…
Delhi Technological University（德里理工大学）；École de technologie supérieure（高等技术学院）
Physical systems whose dynamics are governed by partial differential equations (PDEs) find applications in numerous fields, from engineering design to weather forecasting. The process of obtaining the solution from such PDEs may be computationally expensive for large-scale and parameterized problems. In this work, deep learning techniques developed especially for time-series forecasts, such as LSTM and TCN, or for spatial-feature extraction such as CNN, are employed to model the system dynamics for advection dominated problems. These models take as input a sequence of high-fidelity vector solu…
A Hybrid PCA-PR-Seq2Seq-Adam-LSTM Framework for Time-Series Power Outage Prediction
Graduate School of Arts and Sciences, Columbia University（哥伦比亚大学艺术与科学研究生院）；Sylhet Agricultural University（西利het农业大学）；Department of Electrical Engineering, Columbia University（哥伦比亚大学电气工程系）
本文提出一种融合 PCA、泊松回归、Seq2Seq 和 Adam 优化 LSTM 的混合深度学习框架，用于提升时间序列停电预测的精度与鲁棒性，并在密歇根州真实停电数据上验证其优于现有方法。
Advances in Land Surface Model-based Forecasting: A comparative study of LSTM, Gradient Boosting, and Feedforward Neural Network Models as prognostic state emulators
University of Freiburg（弗莱堡大学）；European Centre for Medium-Range Weather Forecasts（欧洲中期天气预报中心）
Reliable and accurate wind speed prediction has significant impact in many industrial sectors such as economic, business and management among others. This paper presents a new model for wind speed prediction based on Graph Attention Networks (GAT). In particular, the proposed model extends GAT architecture by equipping it with a learnable adjacency matrix as well as incorporating a new attention mechanism with the aim of obtaining attention scores per weather variable. The output of the GAT based model is combined with the LSTM layer in order to exploit both the spatial and temporal characteri…
Balancing Computational Efficiency and Forecast Error in Machine Learning-based Time-Series Forecasting: Insights from Live Experiments on Meteorological Nowcasting
