Probabilistic Multi-Step-Ahead Short-Term Water Demand Forecasting with Lasso
Jens Kley-Holsteg, Florian Ziel
备注：
accepted for publication in ASCE's Journal of Water Resources Planning and Management. Kley-Holsteg, J., Ziel, F.. Forthcoming. "Probabilistic multi-step-ahead short-term water demand forecasting with lasso." Journal of Water Resources Planning and Management. DOI: 10.1061/(ASCE)WR.1943-5452.0001268
Water demand is a highly important variable for operational control and decision making. Hence, the development of accurate forecasts is a valuable field of research to further improve the efficiency of water utilities. Focusing on probabilistic multi-step-ahead forecasting, a time series model is introduced, to capture typical autoregressive, calendar and seasonal effects, to account for time-varying variance, and to quantify the uncertainty and path-dependency of the water demand process. To deal with the high complexity of the water demand process a high-dimensional feature space is applied, which is efficiently tuned by an automatic shrinkage and selection operator (lasso). It allows to obtain an accurate, simple interpretable and fast computable forecasting model, which is well suited for real-time applications. The complete probabilistic forecasting framework allows not only for simulating the mean and the marginal properties, but also the correlation structure between hours within the forecasting horizon. For practitioners, complete probabilistic multi-step-ahead forecasts are of considerable relevance as they provide additional information about the expected aggregated or cumulative water demand, so that a statement can be made about the probability with which a water storage capacity can guarantee the supply over a certain period of time. This information allows to better control storage capacities and to better ensure the smooth operation of pumps. To appropriately evaluate the forecasting performance of the considered models, the energy score (ES) as a strictly proper multidimensional evaluation criterion, is introduced. The methodology is applied to the hourly water demand data of a German water supplier.
Université Libre de Bruxelles（布鲁塞尔自由大学）；Cairo University（开罗大学）
Multi-step ahead forecasting is still an open challenge in time series forecasting. Several approaches that deal with this complex problem have been proposed in the literature but an extensive comparison on a large number of tasks is still missing. This paper aims to fill this gap by reviewing existing strategies for multi-step ahead forecasting and comparing them in theoretical and practical terms. To attain such an objective, we performed a large scale comparison of these different strategies using a large experimental benchmark (namely the 111 series from the NN5 forecasting competition). I…
Modeling price risks is crucial for economic decision making in energy markets. Besides the risk of a single price, the dependence structure of multiple prices is often relevant. We therefore propose a generic and easy-to-implement method for creating multivariate probabilistic forecasts based on univariate point forecasts of day-ahead electricity prices. While each univariate point forecast refers to one of the day's 24 hours, the multivariate forecast distribution models dependencies across hours. The proposed method is based on simple copula techniques and an optional time series component.…
House of Energy Markets and Finance（能源市场与金融研究院）；University of Duisburg-Essen（杜伊斯堡-埃森大学）
We present a winning method of the IEEE DataPort Competition on Day-Ahead Electricity Demand Forecasting: Post-COVID Paradigm. The day-ahead load forecasting approach is based on online forecast combination of multiple point prediction models. It contains four steps: i) data cleaning and preprocessing, ii) a holiday adjustment procedure, iii) training of individual forecasting models, iv) forecast combination by smoothed Bernstein Online Aggregation (BOA). The approach is flexible and can quickly adopt to new energy system situations as they occurred during and after COVID-19 shutdowns. The po…
In the context of capacity planning, forecasting the evolution of informatics servers usage enables companies to better manage their computational resources. We address this problem by collecting key indicator time series and propose to forecast their evolution a day-ahead. Our method assumes that data is structured by a daily seasonality, but also that there is typical evolution of indicators within a day. Then, it uses the combination of a clustering algorithm and Markov Models to produce day-ahead forecasts. Our experiments on real datasets show that the data satisfies our assumption and th…
We present the winning strategy of an electricity demand forecasting competition. This competition was organized to design new forecasting methods for unstable periods such as the one starting in Spring 2020. We rely on state-space models to adapt standard statistical and machine learning models. We claim that it achieves the right compromise between two extremes. On the one hand, purely time-series models such as autoregressives are adaptive in essence but fail to capture dependence to exogenous variables. On the other hand, machine learning methods allow to learn complex dependence to explan…
Électricité de France R&D（法国电力公司研发部）；Viking Conseil（维京咨询公司）
We focus on day-ahead electricity load forecasting of substations of the distribution network in France; therefore, our problem lies between the instability of a single consumption and the stability of a countrywide total demand. Moreover, we are interested in forecasting the loads of over one thousand substations; consequently, we are in the context of forecasting multiple time series. To that end, we rely on an adaptive methodology that provided excellent results at a national scale; the idea is to combine generalized additive models with state-space representations. However, the extension o…
Recurrent and convolutional neural networks are the most common architectures used for time series forecasting in deep learning literature. These networks use parameter sharing by repeating a set of fixed architectures with fixed parameters over time or space. The result is that the overall architecture is time-invariant (shift-invariant in the spatial domain) or stationary. We argue that time-invariance can reduce the capacity to perform multi-step-ahead forecasting, where modelling the dynamics at a range of scales and resolutions is required. We propose ForecastNet which uses a deep feed-fo…
National Technical University of Athens（雅典国家技术大学）；Hellenic Air Force（希腊空军）
Delivering useful hydrological forecasts is critical for urban and agricultural water management, hydropower generation, flood protection and management, drought mitigation and alleviation, and river basin planning and management, among others. In this work, we present and appraise a new simple and flexible methodology for hydrological time series forecasting. This methodology relies on (a) at least two individual forecasting methods and (b) the median combiner of forecasts. The appraisal is made by using a big dataset consisted of 90-year-long mean annual river flow time series from approxima…
Postprocessing of point predictions for probabilistic forecasting of day-ahead electricity prices: The benefits of using isotonic distributional regression
We examine the problem of modeling and forecasting European Day-Ahead and Month-Ahead natural gas prices. For this, we propose two distinct probabilistic models that can be utilized in risk- and portfolio management. We use daily pricing data ranging from 2011 to 2020. Extensive descriptive data analysis shows that both time series feature heavy tails, conditional heteroscedasticity, and show asymmetric behavior in their differences. We propose state-space time series models under skewed, heavy-tailed distributions to capture all stylized facts of the data. They include the impact of autocorre…
Forecasting with Deep Learning: Beyond Average of Average of Average Performance
Faculdade de Engenharia da Universidade do Porto（波尔图大学工程学院）；Laboratory for Artificial Intelligence and Computer Science (LIACC)（人工智能与计算机科学实验室（LIACC））；Fraunhofer Portugal AICOS（弗劳恩霍夫葡萄牙AICOS）
When modeling sub-national mortality rates, it is important to incorporate any possible correlation among sub-populations to improve forecast accuracy. Moreover, forecasts at the sub-national level should aggregate consistently across the forecasts at the national level. In this study, we apply a grouped multivariate functional time series to forecast Australian regional and remote age-specific mortality rates and reconcile forecasts in a group structure using various methods. Our proposed method compares favorably to a grouped univariate functional time series forecasting method by comparing …
We present a regime-switching vector-autoregressive method for very-short-term wind speed forecasting at multiple locations with regimes based on large-scale meteorological phenomena. Statistical methods short-term wind forecasting out-perform numerical weather prediction for forecast horizons up to a few hours, and the spatio-temporal interdependency between geographically dispersed locations may be exploited to improve forecast skill. Here we show that conditioning spatio-temporal interdependency on `atmospheric modes' can further improve forecast performance. The modes are defined from the …
We present new Bayesian methodology for consumer sales forecasting. With a focus on multi-step ahead forecasting of daily sales of many supermarket items, we adapt dynamic count mixture models to forecast individual customer transactions, and introduce novel dynamic binary cascade models for predicting counts of items per transaction. These transactions-sales models can incorporate time-varying trend, seasonal, price, promotion, random effects and other outlet-specific predictors for individual items. Sequential Bayesian analysis involves fast, parallel filtering on sets of decoupled items and…
Lung tumor tracking for radiotherapy requires real-time, multiple-step ahead forecasting of a quasi-periodic time series recording instantaneous tumor locations. We introduce a location-mixture autoregressive (LMAR) process that admits multimodal conditional distributions, fast approximate inference using the EM algorithm and accurate multiple-step ahead predictive distributions. LMAR outperforms several commonly used methods in terms of out-of-sample prediction accuracy using clinical data from lung tumor patients. With its superior predictive performance and real-time computation, the LMAR m…
We improve upon the two-stage sparse vector autoregression (sVAR) method in Davis et al. (2016) by proposing an alternative two-stage modified sVAR method which relies on time series graphical lasso to estimate sparse inverse spectral density in the first stage, and the second stage refines non-zero entries of the AR coefficient matrices using a false discovery rate (FDR) procedure. Our method has the advantage of avoiding the inversion of the spectral density matrix but has to deal with optimization over Hermitian matrices with complex-valued entries. It significantly improves the computation…
Birla Institute of Technology and Science Pilani（比拉理工学院与科学大学皮拉尼分校）
The problem of automatic and accurate forecasting of time-series data has always been an interesting challenge for the machine learning and forecasting community. A majority of the real-world time-series problems have non-stationary characteristics that make the understanding of trend and seasonality difficult. Our interest in this paper is to study the applicability of the popular deep neural networks (DNN) as function approximators for non-stationary TSF. We evaluate the following DNN models: Multi-layer Perceptron (MLP), Convolutional Neural Network (CNN), and RNN with Long-Short Term Memor…
Huazhong University of Science and Technology（华中科技大学）
Despite the superiority of convolutional neural networks demonstrated in time series modeling and forecasting, it has not been fully explored on the design of the neural network architecture and the tuning of the hyper-parameters. Inspired by the incremental construction strategy for building a random multilayer perceptron, we propose a novel Error-feedback Stochastic Modeling (ESM) strategy to construct a random Convolutional Neural Network (ESM-CNN) for time series forecasting task, which builds the network architecture adaptively. The ESM strategy suggests that random filters and neurons of…
Age-specific mortality rates are often disaggregated by different attributes, such as sex, state, ethnic group and socioeconomic status. In making social policies and pricing annuity at national and subnational levels, it is important not only to forecast mortality accurately, but also to ensure that forecasts at the subnational level add up to the forecasts at the national level. This motivates recent developments in grouped functional time series methods (Shang and Hyndman, 2017) to reconcile age-specific mortality forecasts. We extend these grouped functional time series forecasting methods…
Accurate forecasting of passenger flow (i.e., ridership) is critical to the operation of urban metro systems. Previous studies mainly model passenger flow as time series by aggregating individual trips and then perform forecasting based on the values in the past several steps. However, this approach essentially overlooks the fact that passenger flow consists of trips from each individual traveler. For example, a traveler's work trip in the morning can help predict his/her home trip in the evening, while this causal structure cannot be explicitly encoded in standard time series models. In this …
The need to forecast solar irradiation at a specific location over short-time horizons has acquired immense importance. In this paper, we report on analyses results involving statistical and machine learning techniques to predict hourly horizontal solar irradiation at one-hour ahead horizon using data sets from three different cities in the U.S. with different climatic conditions. A simple forecast approach that assumes consecutive days are identical serves as a baseline model against which to compare competing forecast alternatives. One approach is to use seasonal ARIMA models. Surprisingly, …
Numerous deep learning architectures have been developed to accommodate the diversity of time series datasets across different domains. In this article, we survey common encoder and decoder designs used in both one-step-ahead and multi-horizon time series forecasting -- describing how temporal information is incorporated into predictions by each model. Next, we highlight recent developments in hybrid deep learning models, which combine well-studied statistical models with neural network components to improve pure methods in either category. Lastly, we outline some ways in which deep learning c…
We present a methodology for probabilistic load forecasting that is based on lasso (least absolute shrinkage and selection operator) estimation. The model considered can be regarded as a bivariate time-varying threshold autoregressive(AR) process for the hourly electric load and temperature. The joint modeling approach incorporates the temperature effects directly, and reflects daily, weekly, and annual seasonal patterns and public holiday effects. We provide two empirical studies, one based on the probabilistic load forecasting track of the Global Energy Forecasting Competition 2014 (GEFCom20…
The liberalization of electricity markets and the development of renewable energy sources has led to new challenges for decision makers. These challenges are accompanied by an increasing uncertainty about future electricity price movements. The increasing amount of papers, which aim to model and predict electricity prices for a short period of time provided new opportunities for market participants. However, the electricity price literature seem to be very scarce on the issue of medium- to long-term price forecasting, which is mandatory for investment and political decisions. Our paper closes …
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
Obtaining accurate information about future traffic flows of all links in a traffic network is of great importance for traffic management and control applications. This research studies two particular problems in traffic forecasting: (1) capture the dynamic and non-local spatial correlation between traffic links and (2) model the dynamics of temporal dependency for accurate multiple steps ahead predictions. To address these issues, we propose a deep learning framework named Spatial-Temporal Sequence to Sequence model (STSeq2Seq). This model builds on sequence to sequence (seq2seq) architecture…
When modeling sub-national mortality rates, we should consider three features: (1) how to incorporate any possible correlation among sub-populations to potentially improve forecast accuracy through multi-population joint modeling; (2) how to reconcile sub-national mortality forecasts so that they aggregate adequately across various levels of a group structure; (3) among the forecast reconciliation methods, how to combine their forecasts to achieve improved forecast accuracy. To address these issues, we introduce an extension of grouped univariate functional time series method. We first conside…
Volatility forecasts are key inputs in financial analysis. While lasso based forecasts have shown to perform well in many applications, their use to obtain volatility forecasts has not yet received much attention in the literature. Lasso estimators produce parsimonious forecast models. Our forecast combination approach hedges against the risk of selecting a wrong degree of model parsimony. Apart from the standard lasso, we consider several lasso extensions that account for the dynamic nature of the forecast model. We apply forecast combined lasso estimators in a comprehensive forecasting exerc…
In our paper we analyze the relationship between the day-ahead electricity price of the Energy Exchange Austria (EXAA) and other day-ahead electricity prices in Europe. We focus on markets, which settle their prices after the EXAA, which enables traders to include the EXAA price into their calculations. For each market we employ econometric models to incorporate the EXAA price and compare them with their counterparts without the price of the Austrian exchange. By employing a forecasting study, we find that electricity price models can be improved when EXAA prices are considered.
We conduct an extensive empirical study on short-term electricity price forecasting (EPF) to address the long-standing question if the optimal model structure for EPF is univariate or multivariate. We provide evidence that despite a minor edge in predictive performance overall, the multivariate modeling framework does not uniformly outperform the univariate one across all 12 considered datasets, seasons of the year or hours of the day, and at times is outperformed by the latter. This is an indication that combining advanced structures or the corresponding forecasts from both modeling approache…
Day Ahead Electricity Markets (DAMs) in India are thin but growing. Consistent price forecasts are important for their utilization in portfolio optimization models. Univariate or multivariate models with standard exogenous variables such as special day effects etc. are not always useful. Drivers of demand and supply include weather variations over large geographic areas, outages of power system elements and sudden changes in contracts which lead the players to access power exchanges. These needs to be considered in forecasting models. Such models are observed to considerably reduce forecasting…
This paper proposes a regression market for wind agents to monetize data traded among themselves for wind power forecasting. Existing literature on data markets often treats data disclosure as a binary choice or modulates the data quality based on the mismatch between the offer and bid prices. As a result, the market disadvantages either the data sellers due to the overestimation of their willingness to disclose data, or the data buyers due to the lack of useful data being provided. Our proposed regression market determines the data payment based on the least absolute shrinkage and selection o…
Accurate wind power forecasts depend on reliable wind speed forecasts. Numerical Weather Predictions (NWPs) utilize huge amounts of computing time, but still have rather low spatial and temporal resolution. However, stochastic wind speed forecasts perform well in rather high temporal resolution settings. They consume comparably little computing resources and return reliable forecasts, if forecasting horizons are not too long. In the recent literature, spatial interdependence is increasingly taken into consideration. In this paper we propose a new and quite flexible multivariate model that acco…
Wrocław University of Science and Technology（弗罗茨瓦夫科技大学）；Delft University of Technology（代尔夫特理工大学）；Energyville（能源城）；Flemish Institute for Technological Research (VITO)（弗拉芒技术研究院）
Recent advancements in the fields of artificial intelligence and machine learning methods resulted in a significant increase of their popularity in the literature, including electricity price forecasting. Said methods cover a very broad spectrum, from decision trees, through random forests to various artificial neural network models and hybrid approaches. In electricity price forecasting, neural networks are the most popular machine learning method as they provide a non-linear counterpart for well-tested linear regression models. Their application, however, is not straightforward, with multipl…
The Evolution of Probabilistic Price Forecasting Techniques: A Review of the Day-Ahead, Intra-Day, and Balancing Markets
SFI CRT in Artificial Intelligence, School of Computer Science \& IT, University College Cork, Ireland；SFI Insight Centre for Data Analytics, School of Computer Science \& IT, University College Cork, Ireland
Delft University of Technology（代尔夫特理工大学）；VITO-Energyville（VITO-能源城）；Vrije Universiteit Brussel（布鲁塞尔自由大学）
Motivated by the increasing integration among electricity markets, in this paper we propose two different methods to incorporate market integration in electricity price forecasting and to improve the predictive performance. First, we propose a deep neural network that considers features from connected markets to improve the predictive accuracy in a local market. To measure the importance of these features, we propose a novel feature selection algorithm that, by using Bayesian optimization and functional analysis of variance, evaluates the effect of the features on the algorithm performance. In…
Delft University of Technology（代尔夫特理工大学）；Energyville（能源城（能源研究机构））；Flemish Institute for Technological Research (VITO)（弗拉芒技术研究院）；Wrocław University of Science and Technology（弗罗茨瓦夫理工大学）
While the field of electricity price forecasting has benefited from plenty of contributions in the last two decades, it arguably lacks a rigorous approach to evaluating new predictive algorithms. The latter are often compared using unique, not publicly available datasets and across too short and limited to one market test samples. The proposed new methods are rarely benchmarked against well established and well performing simpler models, the accuracy metrics are sometimes inadequate and testing the significance of differences in predictive performance is seldom conducted. Consequently, it is n…
We discuss a concept denoted as Conformal Prediction (CP) in this paper. While initially stemming from the world of machine learning, it was never applied or analyzed in the context of short-term electricity price forecasting. Therefore, we elaborate the aspects that render Conformal Prediction worthwhile to know and explain why its simple yet very efficient idea has worked in other fields of application and why its characteristics are promising for short-term power applications as well. We compare its performance with different state-of-the-art electricity price forecasting models such as qua…
Droughts are a recurring hazard in sub-Saharan Africa, that can wreak huge socioeconomic costs.Acting early based on alerts provided by early warning systems (EWS) can potentially provide substantial mitigation, reducing the financial and human cost. However, existing EWS tend only to monitor current, rather than forecast future, environmental and socioeconomic indicators of drought, and hence are not always sufficiently timely to be effective in practice. Here we present a novel method for forecasting satellite-based indicators of vegetation condition. Specifically, we focused on the 3-month …
There is increasing interest in very short-term and higher-resolution wind power forecasting (from minutes to hours ahead), especially offshore. Statistical methods are of utmost relevance, since weather forecasts cannot be informative for those lead times. Those approaches ought to account for the fact that wind power generation as a stochastic process is nonstationary, double-bounded (by zero and the nominal power of the turbine) and non-linear. Accommodating those aspects may lead to improving both point and probabilistic forecasts. We propose here to focus on generalized logit-normal distr…
We propose a multivariate elastic net regression forecast model for German quarter-hourly electricity spot markets. While the literature is diverse on day-ahead prediction approaches, both the intraday continuous and intraday call-auction prices have not been studied intensively with a clear focus on predictive power. Besides electricity price forecasting, we check for the impact of early day-ahead (DA) EXAA prices on intraday forecasts. Another novelty of this paper is the complementary discussion of economic benefits. A precise estimation is worthless if it cannot be utilized. We elaborate p…
The reliable estimation of forecast uncertainties is crucial for risk-sensitive optimal decision making. In this paper, we propose implicit generative ensemble post-processing, a novel framework for multivariate probabilistic electricity price forecasting. We use a likelihood-free implicit generative model based on an ensemble of point forecasting models to generate multivariate electricity price scenarios with a coherent dependency structure as a representation of the joint predictive distribution. Our ensemble post-processing method outperforms well-established model combination benchmarks. …
Electricity price forecasting (EPF) is a branch of forecasting on the interface of electrical engineering, statistics, computer science, and finance, which focuses on predicting prices in wholesale electricity markets for a whole spectrum of horizons. These range from a few minutes (real-time/intraday auctions and continuous trading), through days (day-ahead auctions), to weeks, months or even years (exchange and over-the-counter traded futures and forward contracts). Over the last 25 years, various methods and computational tools have been applied to intraday and day-ahead EPF. Until the earl…
Inflow forecasts play an essential role in the management of hydropower reservoirs. Forecasts help operators schedule power generation in advance to maximise economic value, mitigate downstream flood risk, and meet environmental requirements. The horizon of operational inflow forecasts is often limited in range to ~2 weeks ahead, marking the predictability barrier of deterministic weather forecasts. Reliable inflow forecasts in the sub-seasonal to seasonal (S2S) range would allow operators to take proactive action to mitigate risks of adverse weather conditions, thereby improving water managem…
Accurate short-term wind speed forecasting is needed for the rapid development and efficient operation of wind energy resources. This is, however, a very challenging problem. Although on the large scale, the wind speed is related to atmospheric pressure, temperature, and other meteorological variables, no improvement in forecasting accuracy was found by incorporating air pressure and temperature directly into an advanced space-time statistical forecasting model, the trigonometric direction diurnal (TDD) model. This paper proposes to incorporate the geostrophic wind as a new predictor in the TD…
