Regression models are essential for a wide range of real-world applications. However, in practice, target values are not always precisely known; instead, they may be represented as intervals of acceptable values. This challenge has led to the development of Interval Regression models. In this study, we provide a comprehensive review of existing Interval Regression models and introduce alternative models for comparative analysis. Experiments are conducted on both real-world and synthetic datasets to offer a broad perspective on model performance. The results demonstrate that no single model is universally optimal, highlighting the importance of selecting the most suitable model for each specific scenario.
Having a regression model, we are interested in finding two-sided intervals that are guaranteed to contain at least a desired proportion of the conditional distribution of the response variable given a specific combination of predictors. We name such intervals predictive intervals. This work presents a new method to find two-sided predictive intervals for non-parametric least squares regression without the homoscedasticity assumption. Our predictive intervals are built by using tolerance intervals on prediction errors in the query point's neighborhood. We proposed a predictive interval model t…
Many machine learning problems can be formulated as predicting labels for a pair of objects. Problems of that kind are often referred to as pairwise learning, dyadic prediction or network inference problems. During the last decade kernel methods have played a dominant role in pairwise learning. They still obtain a state-of-the-art predictive performance, but a theoretical analysis of their behavior has been underexplored in the machine learning literature. In this work we review and unify existing kernel-based algorithms that are commonly used in different pairwise learning settings, ranging f…
We study the generation of prediction intervals in regression for uncertainty quantification. This task can be formalized as an empirical constrained optimization problem that minimizes the average interval width while maintaining the coverage accuracy across data. We strengthen the existing literature by studying two aspects of this empirical optimization. First is a general learning theory to characterize the optimality-feasibility tradeoff that encompasses Lipschitz continuity and VC-subgraph classes, which are exemplified in regression trees and neural networks. Second is a calibration mac…
Conformal Prediction is a framework that produces prediction intervals based on the output from a machine learning algorithm. In this paper we explore the case when training data is made up of multiple parts available in different sources that cannot be pooled. We here consider the regression case and propose a method where a conformal predictor is trained on each data source independently, and where the prediction intervals are then combined into a single interval. We call the approach Non-Disclosed Conformal Prediction (NDCP), and we evaluate it on a regression dataset from the UCI machine l…
By seeking the narrowest prediction intervals (PIs) that satisfy the specified coverage probability requirements, the recently proposed quality-based PI learning principle can extract high-quality PIs that better summarize the predictive certainty in regression tasks, and has been widely applied to solve many practical problems. Currently, the state-of-the-art quality-based PI estimation methods are based on deep neural networks or linear models. In this paper, we propose Highest Density Interval Regression Forest (HDI-Forest), a novel quality-based PI estimation method that is instead based o…
State University of New York（纽约州立大学）；Ninewatt Inc.（Ninewatt公司）；University of Central Asia（中亚大学）
As a result of increasing population and globalization, the demand for energy has greatly risen. Therefore, accurate energy consumption forecasting has become an essential prerequisite for government planning, reducing power wastage and stable operation of the energy management system. In this work we present a comparative analysis of major machine learning models for time series forecasting of household energy consumption. Specifically, we use Weka, a data mining tool to first apply models on hourly and daily household energy consumption datasets available from Kaggle data science community. …
Pattern similarity-based methods are widely used in classification and regression problems. Repeated, similar-shaped cycles observed in seasonal time series encourage to apply these methods for forecasting. In this paper we use the pattern similarity-based methods for forecasting monthly electricity demand expressing annual seasonality. An integral part of the models is the time series representation using patterns of time series sequences. Pattern representation ensures the input and output data unification through trend filtering and variance equalization. Consequently, pattern representatio…
This paper introduces two methods for estimating reliable prediction intervals for local linear least-squares regressions, named Bounded Oscillation Prediction Intervals (BOPI). It also proposes a new measure for comparing interval prediction models named Equivalent Gaussian Standard Deviation (EGSD). The experimental results compare BOPI to other methods using coverage probability, Mean Interval Size and the introduced EGSD measure. The results were generally in favor of the BOPI on considered benchmark regression datasets. It also, reports simulation studies validating the BOPI method's reli…
Comparative Analysis of Data Preprocessing Methods, Feature Selection Techniques and Machine Learning Models for Improved Classification and Regression Performance on Imbalanced Genetic Data
Evergreen Valley College（埃弗格林谷学院）；Gilead Sciences（吉利德科学公司）
Neural interval-censored survival regression with feature selection
Instituto de Ciencias Matemáticas, CSIC（数学科学研究所（西班牙高等科学研究理事会））；Universidad Autónoma de Madrid（马德里自治大学）；Harvard University（哈佛大学）；Université Paris 1 Panthéon-Sorbonne（巴黎第一大学先贤祠-索邦大学）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
Regression Trees for Fast and Adaptive Prediction Intervals
Federal University of São Carlos（圣卡洛斯联邦大学）；Institute of Mathematical Sciences and Computation, University of São Paulo（圣保罗大学数学与计算科学研究所）；Institute of Mathematics and Statistics, University of São Paulo（圣保罗大学数学与统计研究所）
本文提出一种模型无关的方法，通过在 conformity scores 上训练回归树和 Random Forests 来划分特征空间，从而为回归问题校准预测区间，提供局部覆盖保证，并在可扩展性和性能上优于基线。
IBM Research（IBM研究公司）；IBM Research San Jose, CA, USA（美国加利福尼亚州圣何塞IBM研究公司（注：按要求取到公司级别，此处“IBM Research”已明确为公司，无需重复提取））
Data labeling is currently a time-consuming task that often requires expert knowledge. In research settings, the availability of correctly labeled data is crucial to ensure that model predictions are accurate and useful. We propose relatively simple machine learning-based models that achieve high performance metrics in the binary and multiclass classification of radiology reports. We compare the performance of these algorithms to that of a data-driven approach based on NLP, and find that the logistic regression classifier outperforms all other models, in both the binary and multiclass classifi…
Humboldt University of Berlin（柏林洪堡大学）；IÉSEG School of Management（埃塞克高等商学院）；Audencia Business School（南特高等商学院）
Off-the-shelf machine learning algorithms for prediction such as regularized logistic regression cannot exploit the information of time-varying features without previously using an aggregation procedure of such sequential data. However, recurrent neural networks provide an alternative approach by which time-varying features can be readily used for modeling. This paper assesses the performance of neural networks for churn modeling using recency, frequency, and monetary value data from a financial services provider. Results show that RFM variables in combination with LSTM neural networks have la…
Oregon State University（俄勒冈州立大学）；SRI International（SRI国际）
Before delegating a task to an autonomous system, a human operator may want a guarantee about the behavior of the system. This paper extends previous work on conformal prediction for functional data and conformalized quantile regression to provide conformal prediction intervals over the future behavior of an autonomous system executing a fixed control policy on a Markov Decision Process (MDP). The prediction intervals are constructed by applying conformal corrections to prediction intervals computed by quantile regression. The resulting intervals guarantee that with probability $1-δ$ the obser…
Hybrid Gaussian Process Regression with Temporal Feature Extraction for Partially Interpretable Remaining Useful Life Interval Prediction in Aeroengine Prognostics
Federal University of ABC（ABC联邦大学）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）
Symbolic regression is a nonlinear regression method which is commonly performed by an evolutionary computation method such as genetic programming. Quantification of uncertainty of regression models is important for the interpretation of models and for decision making. The linear approximation and so-called likelihood profiles are well-known possibilities for the calculation of confidence and prediction intervals for nonlinear regression models. These simple and effective techniques have been completely ignored so far in the genetic programming literature. In this work we describe the calculat…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Georgia Institute of Technology（佐治亚理工学院）
We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs). INNs are a class of implicit learning models that use implicit equations as layers and have been shown to exhibit several notable benefits over traditional deep neural networks. We first establish that tight inclusion functions of neural networks, which provide the tightest rectangular over-approximation of an input-output map, lead to sharper robustness guarantees than the well-studied robustness measures of local Lipschitz constants. Like Lipschitz constants, tight inclusions functions …
By generating prediction intervals (PIs) to quantify the uncertainty of each prediction in deep learning regression, the risk of wrong predictions can be effectively controlled. High-quality PIs need to be as narrow as possible, whilst covering a preset proportion of real labels. At present, many approaches to improve the quality of PIs can effectively reduce the width of PIs, but they do not ensure that enough real labels are captured. Inductive Conformal Predictor (ICP) is an algorithm that can generate effective PIs which is theoretically guaranteed to cover a preset proportion of data. How…
The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong（香港中文大学）
We propose a nonparametric quantile regression method using deep neural networks with a rectified linear unit penalty function to avoid quantile crossing. This penalty function is computationally feasible for enforcing non-crossing constraints in multi-dimensional nonparametric quantile regression. We establish non-asymptotic upper bounds for the excess risk of the proposed nonparametric quantile regression function estimators. Our error bounds achieve optimal minimax rate of convergence for the Holder class, and the prefactors of the error bounds depend polynomially on the dimension of the pr…
Most of statistics and AI draw insights through modelling discord or variance between sources of information (i.e., inter-source uncertainty). Increasingly, however, research is focusing upon uncertainty arising at the level of individual measurements (i.e., within- or intra-source), such as for a given sensor output or human response. Here, adopting intervals rather than numbers as the fundamental data-type provides an efficient, powerful, yet challenging way forward -- offering systematic capture of uncertainty-at-source, increasing informational capacity, and ultimately potential for insigh…
A Comparative Study of Model Selection Criteria for Symbolic Regression
Aarhus University（奥胡斯大学）；Heuristic and Evolutionary Algorithms Laboratory (HEAL)（启发式和进化算法实验室）；Federal University of ABC（巴西联邦大学ABC分校）；University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学）；University of Modena（摩德纳大学）；Reggio Emilia Department of Physics, Informatics（雷吉奥艾米利亚物理与信息学系）
College of Aerospace Science and Engineering, National University of Defense Technology（国防科技大学航空航天学院）；Defense Innovation Institute, Chinese Academy of Military Science（中国人民解放军军事科学院国防创新研究院）
Temperature field reconstruction is essential for analyzing satellite heat reliability. As a representative machine learning model, the deep convolutional neural network (DCNN) is a powerful tool for reconstructing the satellite temperature field. However, DCNN needs a lot of labeled data to learn its parameters, which is contrary to the fact that actual satellite engineering can only acquire noisy unlabeled data. To solve the above problem, this paper proposes an unsupervised method, i.e., the physics-informed deep Monte Carlo quantile regression method, for reconstructing temperature field a…
Highly accurate interval forecasting of electricity demand is fundamental to the success of reducing the risk when making power system planning and operational decisions by providing a range rather than point estimation. In this study, a novel modeling framework integrating bivariate empirical mode decomposition (BEMD) and support vector regression (SVR), extended from the well-established empirical mode decomposition (EMD) based time series modeling framework in the energy demand forecasting literature, is proposed for interval forecasting of electricity demand. The novelty of this study aris…
Carnegie Mellon University（卡内基梅隆大学）；University of Chicago（芝加哥大学）
Although a majority of the theoretical literature in high-dimensional statistics has focused on settings which involve fully-observed data, settings with missing values and corruptions are common in practice. We consider the problems of estimation and of constructing component-wise confidence intervals in a sparse high-dimensional linear regression model when some covariates of the design matrix are missing completely at random. We analyze a variant of the Dantzig selector [9] for estimating the regression model and we use a de-biasing argument to construct component-wise confidence intervals.…
Fitting high-dimensional statistical models often requires the use of non-linear parameter estimation procedures. As a consequence, it is generally impossible to obtain an exact characterization of the probability distribution of the parameter estimates. This in turn implies that it is extremely challenging to quantify the \emph{uncertainty} associated with a certain parameter estimate. Concretely, no commonly accepted procedure exists for computing classical measures of uncertainty and statistical significance as confidence intervals or $p$-values for these models. We consider here high-dimen…
The high-volume manufacturing of the next generation of semiconductor devices requires advances in measurement signal analysis. Many in the semiconductor manufacturing community have reservations about the adoption of deep learning; they instead prefer other model-based approaches for some image regression problems, and according to the 2021 IEEE International Roadmap for Devices and Systems (IRDS) report on Metrology a SEMI standardization committee may endorse this philosophy. The semiconductor manufacturing community does, however, communicate a need for state-of-the-art statistical analyse…
Highly accurate interval forecasting of a stock price index is fundamental to successfully making a profit when making investment decisions, by providing a range of values rather than a point estimate. In this study, we investigate the possibility of forecasting an interval-valued stock price index series over short and long horizons using multi-output support vector regression (MSVR). Furthermore, this study proposes a firefly algorithm (FA)-based approach, built on the established MSVR, for determining the parameters of MSVR (abbreviated as FA-MSVR). Three globally traded broad market indice…
Interpretable Deep Regression Models with Interval-Censored Failure Time Data
School of Mathematics, Jilin University（吉林大学数学学院）；School of Economics and Statistics, Guangzhou University（广州大学经济与统计学院）；Department of Statistics, Chinese University of Hong Kong（香港中文大学统计系）；School of Data Science, Fudan University（复旦大学数据科学学院）
We develop a method to generate prediction intervals that have a user-specified coverage level across all regions of feature-space, a property called conditional coverage. A typical approach to this task is to estimate the conditional quantiles with quantile regression -- it is well-known that this leads to correct coverage in the large-sample limit, although it may not be accurate in finite samples. We find in experiments that traditional quantile regression can have poor conditional coverage. To remedy this, we modify the loss function to promote independence between the size of the interval…
University of California Irvine（加州大学欧文分校）；London School of Economics and Political Science（伦敦政治经济学院）；North Carolina State University（北卡罗来纳州立大学）
An individualized decision rule (IDR) is a decision function that assigns each individual a given treatment based on his/her observed characteristics. Most of the existing works in the literature consider settings with binary or finitely many treatment options. In this paper, we focus on the continuous treatment setting and propose a jump interval-learning to develop an individualized interval-valued decision rule (I2DR) that maximizes the expected outcome. Unlike IDRs that recommend a single treatment, the proposed I2DR yields an interval of treatment options for each individual, making it mo…
Computer Science Georgia State University Atlanta, USA（计算机科学 乔治亚州立大学 奥斯汀, 美国）；Computer Science Texas Christian University Texas, USA（计算机科学 德克萨斯基督教大学 德克萨斯, 美国）
