In a sequential regression setting, a decision-maker may be primarily concerned with whether the future observation will increase or decrease compared to the current one, rather than the actual value of the future observation. In this context, we introduce the notion of parity calibration, which captures the goal of calibrated forecasting for the increase-decrease (or "parity") event in a timeseries. Parity probabilities can be extracted from a forecasted distribution for the output, but we show that such a strategy leads to theoretical unpredictability and poor practical performance. We then observe that although the original task was regression, parity calibration can be expressed as binary calibration. Drawing on this connection, we use an online binary calibration method to achieve parity calibration. We demonstrate the effectiveness of our approach on real-world case studies in epidemiology, weather forecasting, and model-based control in nuclear fusion.
Machine learning applications often require calibrated predictions, e.g. a 90\% credible interval should contain the true outcome 90\% of the times. However, typical definitions of calibration only require this to hold on average, and offer no guarantees on predictions made on individual samples. Thus, predictions can be systematically over or under confident on certain subgroups, leading to issues of fairness and potential vulnerabilities. We show that calibration for individual samples is possible in the regression setup if the predictions are randomized, i.e. outputting randomized credible …
Online Irregular Multivariate Time Series Forecasting via Uncertainty-Driven Dual-Expert Calibration
Key Laboratory of Big Data \& Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education School of Computer Science；Technology, Beijing Jiaotong University Beijing China；School of Computer Science；Tangshan Research Institute, Beijing Jiaotong University Tangshan China；Key Laboratory of Big Data \& Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education；Technology, Beijing Jiaotong University；Tangshan Research Institute, Beijing Jiaotong University
The climatic challenges are rising across the globe in general and in worst hit under-developed countries in particular. The need for accurate measurements and forecasting of pollutants with low-cost deployment is more pertinent today than ever before. Low-cost air quality monitoring sensors are prone to erroneous measurements, frequent downtimes, and uncertain operational conditions. Such a situation demands a prudent approach to ensure an effective and flexible calibration scheme. We propose MAQ-CaF, a modular air quality calibration, and forecasting methodology, that side-steps the challeng…
Stanford University（斯坦福大学）；Open Climate Fix（开放气候修复）；National Center for Atmospheric Research（美国国家大气研究中心）
Advancing probabilistic solar forecasting methods is essential to supporting the integration of solar energy into the electricity grid. In this work, we develop a variety of state-of-the-art probabilistic models for forecasting solar irradiance. We investigate the use of post-hoc calibration techniques for ensuring well-calibrated probabilistic predictions. We train and evaluate the models using public data from seven stations in the SURFRAD network, and demonstrate that the best model, NGBoost, achieves higher performance at an intra-hourly resolution than the best benchmark solar irradiance …
Georgia Institute of Technology（佐治亚理工学院）；College of Computing（计算学院）
Probabilistic time-series forecasting enables reliable decision making across many domains. Most forecasting problems have diverse sources of data containing multiple modalities and structures. Leveraging information as well as uncertainty from these data sources for well-calibrated and accurate forecasts is an important challenging problem. Most previous work on multi-modal learning and forecasting simply aggregate intermediate representations from each data view by simple methods of summation or concatenation and do not explicitly model uncertainty for each data-view. We propose a general pr…
Departments of Computer Science & Engineering and Data Science, New York University（纽约大学计算机科学与工程系和数据科学系）；Department of Electrical Engineering and Computer Sciences, University of California, Berkeley（加州大学伯克利分校电气工程与计算机科学系）
CoCopula: Conformal Calibration and Copula Aggregation for Reliable Renewable Energy Forecasting
NSF Artificial Intelligence Institute for Advances in Optimization（NSF人工智能优化研究所）；H. Milton Stewart School of Industrial and Systems Engineering（H. Milton Stewart工业与系统工程学院）；Georgia Institute of Technology（佐治亚理工学院）
Google Research（谷歌研究院）；The Hebrew University（希伯来大学）；Technion - Israel Institute of Technology（以色列理工学院）；Bar-Ilan University（巴伊兰大学）；Google Crisis Response（谷歌危机响应团队）
Effective riverine flood forecasting at scale is hindered by a multitude of factors, most notably the need to rely on human calibration in current methodology, the limited amount of data for a specific location, and the computational difficulty of building continent/global level models that are sufficiently accurate. Machine learning (ML) is primed to be useful in this scenario: learned models often surpass human experts in complex high-dimensional scenarios, and the framework of transfer or multitask learning is an appealing solution for leveraging local signals to achieve improved global per…
National University of Sciences and Technology（国立科技大学）；Information Technology University (ITU)（信息技术大学（ITU））
Wheat is the main agricultural crop of Pakistan and is a staple food requirement of almost every Pakistani household making it the main strategic commodity of the country whose availability and affordability is the government's main priority. Wheat food availability can be vastly affected by multiple factors included but not limited to the production, consumption, financial crisis, inflation, or volatile market. The government ensures food security by particular policy and monitory arrangements, which keeps up purchase parity for the poor. Such arrangements can be made more effective if a dyna…
Institute for Information Transmission Problems, Russian Academy of Sciences（俄罗斯科学院信息传输问题研究所）
Using the game-theoretic framework for probability, Vovk and Shafer. have shown that it is always possible, using randomization, to make sequential probability forecasts that pass any countable set of well-behaved statistical tests. This result generalizes work by other authors, who consider only tests of calbration. We complement this result with a lower bound. We show that Vovk and Shafer's result is valid only when the forecasts are computed with unrestrictedly increasing degree of accuracy. When some level of discreteness is fixed, we present a game-theoretic generalization of Oakes' examp…
University of Pennsylvania（宾夕法尼亚大学）；Amazon（亚马逊）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Calibration means that forecasts and average realized frequencies are close. We develop the concept of forecast hedging, which consists of choosing the forecasts so as to guarantee that the expected track record can only improve. This yields all the calibration results by the same simple basic argument while differentiating between them by the forecast-hedging tools used: deterministic and fixed point based versus stochastic and minimax based. Additional contributions are an improved definition of continuous calibration, ensuing game dynamics that yield Nash equilibria in the long run, and a n…
Forecasting high-dimensional time series plays a crucial role in many applications such as demand forecasting and financial predictions. Modern datasets can have millions of correlated time-series that evolve together, i.e they are extremely high dimensional (one dimension for each individual time-series). There is a need for exploiting global patterns and coupling them with local calibration for better prediction. However, most recent deep learning approaches in the literature are one-dimensional, i.e, even though they are trained on the whole dataset, during prediction, the future forecast f…
We consider a general class of forecasting protocols, called "linear protocols", and discuss several important special cases, including multi-class forecasting. Forecasting is formalized as a game between three players: Reality, whose role is to generate observations; Forecaster, whose goal is to predict the observations; and Skeptic, who tries to make money on any lack of agreement between Forecaster's predictions and the actual observations. Our main mathematical result is that for any continuous strategy for Skeptic in a linear protocol there exists a strategy for Forecaster that does not a…
Probabilistic Multi-Regional Solar Power Forecasting with Any-Quantile Recurrent Neural Networks
Faculty of Electrical Engineering, Czestochowa University of Technology（捷斯托霍瓦技术大学电气工程学院）；Faculty of Mathematics and Computer Science, University of Łódź（卢布林大学数学与计算机科学学院）；CAMINO - Centre for Data Analysis, Modelling and Computational Sciences, University of Łódź（卢布林大学CAMINO数据与建模计算科学中心）
ROAR: Retrieval Opportunity-Aware Refinement for Zero-Shot Time Series Forecasting
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；City St George’s, University of London（伦敦大学城圣乔治学院）；Systems Research Institute, Polish Academy of Sciences（波兰科学院系统研究所）
University of Pennsylvania（宾夕法尼亚大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Amazon Inc（亚马逊公司）
We propose to smooth out the calibration score, which measures how good a forecaster is, by combining nearby forecasts. While regular calibration can be guaranteed only by randomized forecasting procedures, we show that smooth calibration can be guaranteed by deterministic procedures. As a consequence, it does not matter if the forecasts are leaked, i.e., made known in advance: smooth calibration can nevertheless be guaranteed (while regular calibration cannot). Moreover, our procedure has finite recall, is stationary, and all forecasts lie on a finite grid. To construct the procedure, we deal…
We propose a novel methodology for estimating the epidemiological parameters of a modified SIRD model (acronym of Susceptible, Infected, Recovered and Deceased individuals) and perform a short-term forecast of SARS-CoV-2 virus spread. We mainly focus on forecasting number of deceased. The procedure was tested on reported data for Poland. For some short-time intervals we performed numerical test investigating stability of parameter estimates in the proposed approach. Numerical experiments confirm the effectiveness of short-term forecasts (up to 2 weeks) and stability of the method. To improve t…
Improving irradiance forecasting is critical to further increase the share of solar in the energy mix. On a short time scale, fish-eye cameras on the ground are used to capture cloud displacements causing the local variability of the electricity production. As most of the solar radiation comes directly from the Sun, current forecasting approaches use its position in the image as a reference to interpret the cloud cover dynamics. However, existing Sun tracking methods rely on external data and a calibration of the camera, which requires access to the device. To address these limitations, this s…
Towards Instance-Wise Calibration: Local Amortized Diagnostics and Reshaping of Conditional Densities (LADaR)
1 Department of Physics；Astronomy, University of Pittsburgh；Cosmology Center (PITT PACC), University of Pittsburgh；3 Department of Statistical Sciences, University of Toronto；4 Canadian Institute for Theoretical Astrophysics (CITA), University of Toronto；5 Dunlap Institute for Astronomy \& Astrophysics, University of Toronto；6 Vector Institute；7 Department of Statistics；Data Science, Carnegie Mellon University；8 Department of Statistics, Federal University of S\ a o Carlos (UFSCar)
LoFT-LLM: Low-Frequency Time-Series Forecasting with Large Language Models
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（计算机科学与技术学院，哈尔滨工业大学（深圳））；The Chinese University of Hong Kong（香港中文大学）
