The soft-margin support vector machine (SVM) is a ubiquitous tool for prediction of binary-response data. However, the SVM is characterized entirely via a numerical optimization problem, rather than a probability model, and thus does not directly generate probabilistic inferential statements as outputs. We consider a probabilistic regression model for binary-response data that is based on the optimization problem that characterizes the SVM. Under weak regularity assumptions, we prove that the maximum likelihood estimate (MLE) of our model exists, and that it is consistent and asymptotically normal. We further assess the performance of our model via simulation studies, and demonstrate its use in real data applications regarding spam detection and well water access.
We propose a new class of estimators of the multivariate response linear regression coefficient matrix that exploits the assumption that the response and predictors have a joint multivariate Normal distribution. This allows us to indirectly estimate the regression coefficient matrix through shrinkage estimation of the parameters of the inverse regression, or the conditional distribution of the predictors given the responses. We establish a convergence rate bound for estimators in our class and we study two examples. The first example estimator exploits an assumption that the inverse regression…
In regression models for categorical data a linear model is typically related to the response variables via a transformation of probabilities called the link function. We introduce an approach based on two link functions for binary data named log-mean (LM) and log-mean linear (LML), respectively. The choice of the link function plays a key role for the interpretation of the model, and our approach is especially appealing in terms of interpretation of the effects of covariates on the association of responses. Similarly to Poisson regression, the LM and LML regression coefficients of single outc…
Binary regression models are ubiquitous in virtually every scientific field. Frequently, traditional generalized linear models fail to capture the variability in the probability surface that gives rise to the binary observations and novel methodology is required. This has generated a substantial literature comprised of binary regression models motivated by various applications. We describe a novel organization of generalizations to traditional binary regression methods based on the familiar three-part structure of generalized linear models (random component, systematic component, link function…
Logistic regression model is widely used in many studies to investigate the relationship between a binary response variable $Y$ and a set of potential predictors $\mathbf X$. The binary response may represent, for example, the occurrence of some outcome of interest ($Y=1$ if the outcome occurred and $Y=0$ otherwise). When the dependent variable $Y$ represents a rare event, the logistic regression model shows relevant drawbacks. In order to overcome these drawbacks we propose the Generalized Extreme Value (GEV) regression model. In particular, we suggest the quantile function of the GEV distrib…
Testing procedures for assessing a parametric regression model with circular response and $\mathbb{R}^d$-valued covariate are proposed and analyzed in this work both for independent and for spatially correlated data. The test statistics are based on a circular distance comparing a (non-smoothed or smoothed) parametric circular estimator and a nonparametric one. Properly designed bootstrap procedures for calibrating the tests in practice are also presented. Finite sample performance of the tests in different scenarios with independent and spatially correlated samples, is analyzed by simulations…
Quantile regression permits describing how quantiles of a scalar response variable depend on a set of predictors. Because a unique definition of multivariate quantiles is lacking, extending quantile regression to multivariate responses is somewhat complicated. In this paper, we describe a simple approach based on a two-step procedure: in the first step, quantile regression is applied to each response separately; in the second step, the joint distribution of the signs of the residuals is modeled through multinomial regression. The described approach does not require a multidimensional definitio…
Univariate isotonic regression (IR) has been used for nonparametric estimation in dose-response and dose-finding studies. One undesirable property of IR is the prevalence of piecewise-constant stretches in its estimates, whereas the dose-response function is usually assumed to be strictly increasing. We propose a simple modification to IR, called centered isotonic regression (CIR). CIR's estimates are strictly increasing in the interior of the dose range. In the absence of monotonicity violations, CIR and IR both return the original observations. Numerical examination indicates that for sample…
Hypothesis testing procedures are developed to assess linear operator constraints in function-on-scalar regression when incomplete functional responses are observed. The approach enables statistical inferences about the shape and other aspects of the functional regression coefficients within a unified framework encompassing three incomplete sampling scenarios: (i) partially observed response functions as curve segments over random sub-intervals of the domain; (ii) discretely observed functional responses with additive measurement errors; and (iii) the composition of former two scenarios, where…
We propose a new prediction method for multivariate linear regression problems where the number of features is less than the sample size but the number of outcomes is extremely large. Many popular procedures, such as penalized regression procedures, require parameter tuning that is computationally untenable in such large-scale problems. We take a different approach, motivated by ideas from simultaneous estimation problems, that performs linear shrinkage on ordinary least squares parameter estimates. Our approach is extremely computationally efficient and tuning-free. We show that it can asympt…
We develop a new method to fit the multivariate response linear regression model that exploits a parametric link between the regression coefficient matrix and the error covariance matrix. Specifically, we assume that the correlations between entries in the multivariate error random vector are proportional to the cosines of the angles between their corresponding regression coefficient matrix columns, so as the angle between two regression coefficient matrix columns decreases, the correlation between the corresponding errors increases. We highlight two models under which this parameterization ar…
This paper studies the inference of the regression coefficient matrix under multivariate response linear regressions in the presence of hidden variables. A novel procedure for constructing confidence intervals of entries of the coefficient matrix is proposed. Our method first utilizes the multivariate nature of the responses by estimating and adjusting the hidden effect to construct an initial estimator of the coefficient matrix. By further deploying a low-dimensional projection procedure to reduce the bias introduced by the regularization in the previous step, a refined estimator is proposed …
Statisticians generally use ordinary least squares to minimize the random error in a subject response with respect to independent explanatory variable. However, Wooten shows illustrates how ordinary least squares can be used to minimize the random error in the system without defining a subject response. Using lattice design Wooten shows that non-response analysis is a superior alternative rotation of the pyramidal relationship between random variables and parameter estimates in multi-linear regression. Non-Response Analysis for simple linear co-linearity and Rotational Analysis in Simple Linea…
This article considers nonparametric regression models with multivariate covariates and with responses missing at random. We estimate the regression function with a local polynomial smoother. The residual-based empirical distribution function that only uses complete cases, i.e. residuals that can actually be constructed from the data, is shown to be efficient in the sense of Hájek and Le Cam. In the proofs we derive, more generally, the efficient influence function for estimating an arbitrary linear functional of the error distribution; this covers the distribution function as a special case. …
Generalized extreme value (GEV) regression is often more adapted when we investigate a relationship between a binary response variable $Y$ which represents a rare event and potentiel predictors $\mathbf{X}$. In particular, we use the quantile function of the GEV distribution as link function. Bootstrapping assigns measures of accuracy (bias, variance, confidence intervals, prediction error, test of hypothesis) to sample estimates. This technique allows estimation of the sampling distribution of almost any statistic using random sampling methods. Bootstrapping estimates the properties of an est…
This paper develops a framework for quantile regression in binary longitudinal data settings. A novel Markov chain Monte Carlo (MCMC) method is designed to fit the model and its computational efficiency is demonstrated in a simulation study. The proposed approach is flexible in that it can account for common and individual-specific parameters, as well as multivariate heterogeneity associated with several covariates. The methodology is applied to study female labor force participation and home ownership in the United States. The results offer new insights at the various quantiles, which are of …
We discuss semiparametric regression when only the ranks of responses are observed. The model is $Y_i = F (\mathbf{x}_i'{\boldsymbolβ}_0 + \varepsilon_i)$, where $Y_i$ is the unobserved response, $F$ is a monotone increasing function, $\mathbf{x}_i$ is a known $p-$vector of covariates, ${\boldsymbolβ}_0$ is an unknown $p$-vector of interest, and $\varepsilon_i$ is an error term independent of $\mathbf{x}_i$. We observe $\{(\mathbf{x}_i,R_n(Y_i)) : i = 1,\ldots ,n\}$, where $R_n$ is the ordinal rank function. We explore a novel estimator under Gaussian assumptions. We discuss the literature, ap…
In this paper, we consider binary response models with linear quantile restrictions. Considerably generalizing previous research on this topic, our analysis focuses on an infinite collection of quantile estimators. We derive a uniform linearisation for the properly standardized empirical quantile process and discover some surprising differences with the setting of continuously observed responses. Moreover, we show that considering quantile processes provides an effective way of estimating binary choice probabilities without restrictive assumptions on the form of the link function, heteroskedas…
kNN estimation in semi-functional partial linear regression with missing responses at random
Universidade da Coruña（拉科鲁尼亚大学）；Centro de Investigación en Tecnologías de la Información y las Comunicaciones (CITIC)（信息与通信技术研究中心）；Universidad Carlos III de Madrid（马德里卡洛斯三世大学）
In this paper, we develop two fully parametric quantile regression models, based on power Johnson SB distribution Cancho et al. (2020), for modeling unit interval response at different quantiles. In particular, the conditional distribution is modelled by the power Johnson SB distribution. The maximum likelihood method is employed to estimate the model parameters. Simulation studies are conducted to evaluate the performance of the maximum likelihood estimators in finite samples. Furthermore, we discuss residuals and influence diagnostic tools. The effectiveness of our proposals is illustrated w…
It is well established that migratory birds in general have advanced their arrival times in spring, and in this paper we investigate potential ways of enhancing the level of detail in future phenological analyses. We perform single as well as multiple species analyses, using linear models on empirical quantiles, non-parametric quantile regression and likelihood-based parametric quantile regression with asymmetric Laplace distributed error terms. We conclude that non-parametric quantile regression appears most suited for single as well as multiple species analyses.
In functional linear regression, the parameters estimation involves solving a non necessarily well-posed problem and it has points of contact with a range of methodologies, including statistical smoothing, deconvolution and projection on finite-dimensional subspaces. We discuss the standard approach based explicitly on functional principal components analysis, nevertheless the choice of the number of basis components remains something subjective and not always properly discussed and justified. In this work we discuss inferential properties of least square estimation in this context with differ…
Misclassification of binary responses, if ignored, may severely bias the maximum likelihood estimators (MLE) of regression parameters. For such data, a binary regression model incorporating misclassification probabilities is extensively used by researchers in different application contexts. The model, however, suffers from a serious estimation problem because of confounding of the unknown misclassification probabilities with the regression parameters. To overcome this problem, in addition to the main sample, use of internal validation data is proposed. However, the maximum likelihood estimator…
We consider a multivariate linear response regression in which the number of responses and predictors is large and comparable with the number of observations, and the rank of the matrix of regression coefficients is assumed to be small. We study the distribution of singular values for the matrix of regression coefficients and for the matrix of predicted responses. For both matrices, it is found that the limit distribution of the largest singular value is a rescaling of the Tracy-Widom distribution. Based on this result, we suggest algorithms for the model rank selection and compare them with t…
The multi-cycle organization of modern university systems stimulates the interest in studying the progression to higher level degree courses during the academic career. In particular, after the achievement of the first level qualification (Bachelor degree), students have to decide whether to continue their university studies, by enrolling in a second level (Master) programme, or to conclude their training experience. In this work we propose a binary quantile regression approach to analyze the Bachelor-Master transition phenomenon with the adoption of the Bayesian inferential perspective. In ad…
Data consisting of samples of probability density functions are increasingly prevalent, necessitating the development of methodologies for their analysis that respect the inherent nonlinearities associated with densities. In many applications, density curves appear as functional response objects in a regression model with vector predictors. For such models, inference is key to understand the importance of density-predictor relationships, and the uncertainty associated with the estimated conditional mean densities, defined as conditional Fréchet means under a suitable metric. Using the Wasserst…
SZTAKI: Institute for Computer Science and Control（SZTAKI：计算机科学与控制研究所）；Eötvös Loránd Research Network（厄特沃什·罗兰研究网络）
In this paper we suggest two statistical hypothesis tests for the regression function of binary classification based on conditional kernel mean embeddings. The regression function is a fundamental object in classification as it determines both the Bayes optimal classifier and the misclassification probabilities. A resampling based framework is presented and combined with consistent point estimators of the conditional kernel mean map, in order to construct distribution-free hypothesis tests. These tests are introduced in a flexible manner allowing us to control the exact probability of type I e…
For a regression problem with a binary label response, we examine the problem of constructing confidence intervals for the label probability conditional on the features. In a setting where we do not have any information about the underlying distribution, we would ideally like to provide confidence intervals that are distribution-free---that is, valid with no assumptions on the distribution of the data. Our results establish an explicit lower bound on the length of any distribution-free confidence interval, and construct a procedure that can approximately achieve this length. In particular, thi…
In this paper, we study the detection boundary for minimax hypothesis testing in the context of high-dimensional, sparse binary regression models. Motivated by genetic sequencing association studies for rare variant effects, we investigate the complexity of the hypothesis testing problem when the design matrix is sparse. We observe a new phenomenon in the behavior of detection boundary which does not occur in the case of Gaussian linear regression. We derive the detection boundary as a function of two components: a design matrix sparsity index and signal strength, each of which is a function o…
In this article, we propose some new generalizations of M-estimation procedures for single-index regression models in presence of randomly right-censored responses. We derive consistency and asymptotic normality of our estimates. The results are proved in order to be adapted to a wide range of techniques used in a censored regression framework (e.g. synthetic data or weighted least squares). As in the uncensored case, the estimator of the single-index parameter is seen to have the same asymptotic behavior as in a fully parametric scheme. We compare these new estimators with those based on the …
Mitscherlich's function is a well-known three-parameter non-linear regression function that quantifies the relation between a stimulus or a time variable and a response. Optimal designs for this function have been constructed only for normally distributed responses with homoscedastic variances. In this paper, we construct D-optimal designs for discrete and continuous responses having their distribution function in the exponential family. We also demonstrate the connection with D-optimality for weighted linear regression.
In this paper we present a general convex optimization approach for solving high-dimensional multiple response tensor regression problems under low-dimensional structural assumptions. We consider using convex and weakly decomposable regularizers assuming that the underlying tensor lies in an unknown low-dimensional subspace. Within our framework, we derive general risk bounds of the resulting estimate under fairly general dependence structure among covariates. Our framework leads to upper bounds in terms of two very simple quantities, the \emph{Gaussian width} of a convex set in tensor space a…
We propose an extrinsic regression framework for modeling data with manifold valued responses and Euclidean predictors. Regression with manifold responses has wide applications in shape analysis, neuroscience, medical imaging and many other areas. Our approach embeds the manifold where the responses lie onto a higher dimensional Euclidean space, obtains a local regression estimate in that space, and then projects this estimate back onto the image of the manifold. Outside the regression setting both intrinsic and extrinsic approaches have been proposed for modeling i.i.d manifold-valued data. H…
We consider the double functional nonparametric regression model $Y=r(X)+ε$, where the response variable $Y$ is Hilbert space-valued and the covariate $X$ takes values in a pseudometric space. The data satisfy an ergodicity criterion which dates back to Laib and Louani (2010) and are arranged in a triangular array. So our model also applies to samples obtained from spatial processes, e.g., stationary random fields indexed by the regular lattice $\mathbb{Z}^N$ for some $N\in\mathbb{N}_+$. We consider a kernel estimator of the Nadaraya--Watson type for the regression operator $r$ and study its l…
Distribution-on-distribution regression considers the problem of formulating and estimating a regression relationship where both covariate and response are probability distributions. The optimal transport distributional regression model postulates that the conditional Fréchet mean of the response distribution is linked to the covariate distribution via an optimal transport map. We establish the minimax rate of estimation of such a regression function, by deriving a lower-bound that matches the convergence rate attained by the Fréchet least squares estimator.
A nonparametric and locally adaptive Bayesian estimator is proposed for estimating a binary regression. Flexibility is obtained by modeling the binary regression as a mixture of probit regressions with the argument of each probit regression having a thin plate spline prior with its own smoothing parameter and with the mixture weights depending on the covariates. The estimator is compared to a single spline estimator and to a recently proposed locally adaptive estimator. The methodology is illustrated by applying it to both simulated and real examples.
In binary and ordinal regression one can distinguish between a location component and a scaling component. While the former determines the location within the range of the response categories, the scaling indicates variance heterogeneity. In particular since it has been demonstrated that misleading effects can occur if one ignores the presence of a scaling component it is important to account for potential scaling effects in the regression model, which is not possible in available recursive partitioning methods. The proposed recursive partitioning method yields two trees, one for the location …
Fast variational approximate algorithms are developed for Bayesian semiparametric regression when the response variable is a count, i.e. a non-negative integer. We treat both the Poisson and Negative Binomial families as models for the response variable. Our approach utilizes recently developed methodology known as non-conjugate variational message passing. For concreteness, we focus on generalized additive mixed models, although our variational approximation approach extends to a wide class of semiparametric regression models such as those containing interactions and elaborate random effect s…
RE-EM tree is a tree-based method that combines the regression tree and the linear mixed effects model for modeling univariate response longitudinal or clustered data. In this paper, we generalize the RE-EM tree method to multivariate response data, by adopting the Multivariate Regression Tree method proposed by De'Ath [2002]. The Multivariate RE-EM tree method estimates a population-level single tree structure that is driven by the multiple responses simultaneously and object-level random effects for each response variable, where correlation between the response variables and between the asso…
We investigate R-optimal designs for multi-response regression models with multi-factors, where the random errors in these models are correlated. Several theoretical results are derived for Roptimal designs, including scale invariance, reflection symmetry, line and plane symmetry, and dependence on the covariance matrix of the errors. All the results can be applied to linear and nonlinear models. In addition, an efficient algorithm based on an interior point method is developed for finding R-optimal designs on discrete design spaces. The algorithm is very flexible, and can be applied to any mu…
We propose local polynomial estimators for the conditional mean of a continuous response when only pooled response data are collected under different pooling designs. Asymptotic properties of these estimators are investigated and compared. Extensive simulation studies are carried out to compare finite sample performance of the proposed estimators under various model settings and pooling strategies. We apply the proposed local polynomial regression methods to two real-life applications to illustrate practical implementation and performance of the estimators for the mean function.
In this paper we have proposed a general class of modified regression type estimator in systematic sampling under non-response to estimate the population mean using auxiliary information. The expressions of bias and mean square error (MSE) up to the first order approximations are derived. A numerical study is included to support the theoretical results.
Nonparametric estimators of a regression function with circular response and Rd-valued predictor are considered in this work. Local polynomial type estimators are proposed and studied. Expressions for their asymptotic biases and variances are derived, and some guidelines to select asymptotically local optimal bandwidth matrices are also given. The finite sample behavior of the proposed estimators is assessed through simulations and their performance is also illustrated with a real data set.
For an observed response that is composed by a set - or vector - of positive values that sum up to 1, the Dirichlet distribution (Bol'shev, 2018) is a helpful mathematical construction for the quantification of the data-generating mechanics underlying this process. In applications, these response-sets are usually denoted as proportions, or compositions of proportions, and by means of covariates, one wishes to manifest the underlying signal - by changes in the value of these covariates - leading to differently distributed response compositions. This article gives a brief introduction into this …
In this article, we consider convergence rates in functional linear regression with functional responses, where the linear coefficient lies in a reproducing kernel Hilbert space (RKHS). Without assuming that the reproducing kernel and the covariate covariance kernel are aligned, or assuming polynomial rate of decay of the eigenvalues of the covariance kernel, convergence rates in prediction risk are established. The corresponding lower bound in rates is derived by reducing to the scalar response case. Simulation studies and two benchmark datasets are used to illustrate that the proposed approa…
