We consider the problem of estimation of a shift parameter of an unknown symmetric function in Gaussian white noise. We introduce a notion of semiparametric second-order efficiency and propose estimators that are semiparametrically efficient and second-order efficient in our model. These estimators are of a penalized maximum likelihood type with an appropriately chosen penalty. We argue that second-order efficiency is crucial in semiparametric problems since only the second-order terms in asymptotic expansion for the risk account for the behavior of the ``nonparametric component'' of a semiparametric procedure, and they are not dramatically smaller than the first-order terms.
In this paper, we propose a new semiparametric regression estimator by using a hybrid technique of a parametric approach and a nonparametric penalized spline method. The overall shape of the true regression function is captured by the parametric part, while its residual is consistently estimated by the nonparametric part. Asymptotic theory for the proposed semiparametric estimator is developed, showing that its behavior is dependent on the asymptotics for the nonparametric penalized spline estimator as well as on the discrepancy between the true regression function and the parametric part. As …
We study the Cox models with semiparametric relative risk, which can be partially linear with one nonparametric component, or multiple additive or nonadditive nonparametric components. A penalized partial likelihood procedure is proposed to simultaneously estimate the parameters and select variables for both the parametric and the nonparametric parts. Two penalties are applied sequentially. The first penalty, governing the smoothness of the multivariate nonlinear covariate effect function, provides a smoothing spline ANOVA framework that is exploited to derive an empirical model selection tool…
We suggest an iterative approach to computing K-step maximum likelihood estimates (MLE) of the parametric components in semiparametric models based on their profile likelihoods. The higher order convergence rate of K-step MLE mainly depends on the precision of its initial estimate and the convergence rate of the nuisance functional parameter in the semiparametric model. Moreover, we can show that the K-step MLE is as asymptotically efficient as the regular MLE after a finite number of iterative steps. Our theory is verified for several specific semiparametric models. Simulation studies are als…
A common practice in obtaining a semiparametric efficient estimate is through iteratively maximizing the (penalized) log-likelihood w.r.t. its Euclidean parameter and functional nuisance parameter via Newton-Raphson algorithm. The purpose of this paper is to provide a formula in calculating the minimal number of iterations $k^\ast$ needed to produce an efficient estimate $\hatθ_n^{(k^\ast)}$ from a theoretical point of view. We discover that (a) $k^\ast$ depends on the convergence rates of the initial estimate and nuisance estimate; (b) more than $k^\ast$ iterations, i.e., $k$, will only impro…
The penalized profile sampler for semiparametric inference is an extension of the profile sampler method (Lee, Kosorok and Fine, 2005) obtained by profiling a penalized log-likelihood. The idea is to base inference on the posterior distribution obtained by multiplying a profiled penalized log-likelihood by a prior for the parametric component, where the profiling and penalization are applied to the nuisance parameter. Because the prior is not applied to the full likelihood, the method is not strictly Bayesian. A benefit of this approximately Bayesian method is that it circumvents the need to p…
In this article, we propose a penalized high dimensional semiparametric model average quantile prediction approach that is robust for forecasting the conditional quantile of the response. We consider a two-step estimation procedure. In the first step, we use a local linear regression approach to estimate the individual marginal quantile functions, and approximate the conditional quantile of the response by an affine combination of one-dimensional marginal quantile regression functions. In the second step, based on the nonparametric kernel estimates of the marginal quantile regression functions…
We give an overview of statistical models and likelihood, together with two of its variants: penalized and hierarchical likelihood. The Kullback-Leibler divergence is referred to repeatedly, for defining the misspecification risk of a model, for grounding the likelihood and the likelihood crossvalidation which can be used for choosing weights in penalized likelihood. Families of penalized likelihood and sieves estimators are shown to be equivalent. The similarity of these likelihood with a posteriori distributions in a Bayesian approach is considered.
There is a difficulty in finding an estimate of variance of the profile likelihood estimator in the joint model of longitudinal and survival data. We solve the difficulty by introducing the ``statistical generalized derivative''. The derivative is used to show the asymptotic normality of the estimator without assuming the second derivative of the density function in the model exists.
We consider the problem of simultaneous variable selection and estimation in additive, partially linear models for longitudinal/clustered data. We propose an estimation procedure via polynomial splines to estimate the nonparametric components and apply proper penalty functions to achieve sparsity in the linear part. Under reasonable conditions, we obtain the asymptotic normality of the estimators for the linear components and the consistency of the estimators for the nonparametric components. We further demonstrate that, with proper choice of the regularization parameter, the penalized estimat…
In this work we present a simple estimation procedure for a general frailty model for analysis of prospective correlated failure times. Earlier work showed this method to perform well in a simulation study. Here we provide rigorous large-sample theory for the proposed estimators of both the regression coefficient vector and the dependence parameter, including consistent variance estimators.
The problem of estimating the shift (or, equivalently, the center of symmetry) of an unknown symmetric and periodic function $f$ observed in Gaussian white noise is considered. Using the blockwise Stein method, a penalized profile likelihood with a data-driven penalization is introduced so that the estimator of the center of symmetry is defined as the maximizer of the penalized profile likelihood. This estimator has the advantage of being independent of the functional class to which the signal $f$ si assumed to belong and, furthermore, is shown to be semiparametrically adaptive and efficient. …
Semiparametric regression offers a flexible framework for modeling non-linear relationships between a response and covariates. A prime example are generalized additive models where splines (say) are used to approximate non-linear functional components in conjunction with a quadratic penalty to control for overfitting. Estimation and inference are then generally performed based on the penalized likelihood, or under a mixed model framework. The penalized likelihood framework is fast but potentially unstable, and choosing the smoothing parameters needs to be done externally using cross-validation…
We introduce a covariate-specific total variation penalty in two semiparametric models for the rate function of recurrent event process. The two models are a stratified Cox model, introduced in Prentice et al. (1981), and a stratified Aalen's additive model. We show the consistency and asymptotic normality of our penalized estimators. We demonstrate, through a simulation study, that our estimators outperform classical estimators for small to moderate sample sizes. Finally an application to the bladder tumour data of Byar (1980) is presented.
Penalized methods are applied to quasi likelihood analysis for stochastic differential equation models. In this paper, we treat the quasi likelihood function and the associated statistical random field for which a polynomial type large deviation inequality holds. Then penalty terms do not disturb a polynomial type large deviation inequality. This property ensures the convergence of moments of the associated estimator which plays an important role to evaluate the upper bound of the probability that model selection is incorrect.
In this article, the weighted empirical likelihood is applied to a general setting of two-sample semiparametric models, which includes biased sampling models and case-control logistic regression models as special cases. For various types of censored data, such as right censored data, doubly censored data, interval censored data and partly interval-censored data, the weighted empirical likelihood-based semiparametric maximum likelihood estimator $(\tildeθ_n,\tilde{F}_n)$ for the underlying parameter $θ_0$ and distribution $F_0$ is derived, and the strong consistency of $(\tildeθ_n,\tilde{F}_n)$…
A class of variable selection procedures for parametric models via nonconcave penalized likelihood was proposed by Fan and Li to simultaneously estimate parameters and select important variables. They demonstrated that this class of procedures has an oracle property when the number of parameters is finite. However, in most model selection problems the number of parameters should be large and grow with the sample size. In this paper some asymptotic properties of the nonconcave penalized likelihood are established for situations in which the number of parameters tends to \infty as the sample siz…
We extend the correspondence between two-stage coding procedures in data compression and penalized likelihood procedures in statistical estimation. Traditionally, this had required restriction to countable parameter spaces. We show how to extend this correspondence in the uncountable parameter case. Leveraging the description length interpretations of penalized likelihood procedures we devise new techniques to derive adaptive risk bounds of such procedures. We show that the existence of certain countable coverings of the parameter space implies adaptive risk bounds and thus our theory is quite…
Penalized likelihood methods are fundamental to ultra-high dimensional variable selection. How high dimensionality such methods can handle remains largely unknown. In this paper, we show that in the context of generalized linear models, such methods possess model selection consistency with oracle properties even for dimensionality of Non-Polynomial (NP) order of sample size, for a class of penalized likelihood approaches using folded-concave penalty functions, which were introduced to ameliorate the bias problems of convex penalty functions. This fills a long-standing gap in the literature whe…
During the past few decades, missing-data problems have been studied extensively, with a focus on the ignorable missing case, where the missing probability depends only on observable quantities. By contrast, research into non-ignorable missing data problems is quite limited. The main difficulty in solving such problems is that the missing probability and the regression likelihood function are tangled together in the likelihood presentation, and the model parameters may not be identifiable even under strong parametric model assumptions. In this paper we discuss a semiparametric model for non-ig…
We wish to estimate conditional density using Gaussian Mixture Regression model with logistic weights and means depending on the covariate. We aim at selecting the number of components of this model as well as the other parameters by a penalized maximum likelihood approach. We provide a lower bound on penalty, proportional up to a logarithmic term to the dimension of each model, that ensures an oracle inequality for our estimator. Our theoretical analysis is supported by some numerical experiments.
This work studies the statistical properties of the maximum penalized likelihood approach in a semi-parametric framework. We recall the penalized likelihood approach for estimating a function and review some asymptotic results. We investigate the properties of two estimators of the variance of maximum penalized likelihood estimators: sandwich estimator and a Bayesian estimator. The coverage rates of confidence intervals based on these estimators are studied through a simulation study of survival data. In a first simulation the coverage rates for the survival function and the hazard function ar…
Consider semiparametric models that display local asymptotic exponentiality (Ibragimov and Has'minskii (1981)), an asymptotic property of the likelihood associated with discontinuities of densities. Our interest goes to estimation of the location of such discontinuities while other aspects of the density form a nuisance parameter. It is shown that under certain conditions on model and prior, the posterior distribution displays Bernstein-von Mises-type asymptotic behaviour, with exponential distributions as the limiting sequence. In contrast to regular settings, the maximum likelihood estimator…
Quantitative studies in many fields involve the analysis of multivariate data of diverse types, including measurements that we may consider binary, ordinal and continuous. One approach to the analysis of such mixed data is to use a copula model, in which the associations among the variables are parameterized separately from their univariate marginal distributions. The purpose of this article is to provide a method of semiparametric inference for copula models via the construction of what we call a marginal set likelihood function for the association parameters. The proposed method of inference…
We investigate a semiparametric regression model where one gets noisy non linear non invertible functions of the observations. We focus on the application to bearings-only tracking. We first investigate the least squares estimator and prove its consistency and asymptotic normality under mild assumptions. We study the semiparametric likelihood process and prove local asymptotic normality of the model. This allows to define the efficient Fisher information as a lower bound for the asymptotic variance of regular estimators, and to prove that the parametric likelihood estimator is regular and asym…
In this work we provide a simple estimation procedure for a general frailty model for analysis of prospective correlated failure times. Rigorous large-sample theory for the proposed estimators of both the regression coefficient vector and the dependence parameter is given, including consistent variance estimators. In a simulation study under the widely used gamma frailty model, our proposed approach was found to have essentially the same efficiency as the EM-based estimator considered by other authors, with negligible difference between the standard errors of the two estimators. The proposed a…
Fan and Li propose a family of variable selection methods via penalized likelihood using concave penalty functions. The nonconcave penalized likelihood estimators enjoy the oracle properties, but maximizing the penalized likelihood function is computationally challenging, because the objective function is nondifferentiable and nonconcave. In this article, we propose a new unified algorithm based on the local linear approximation (LLA) for maximizing the penalized likelihood for a broad class of concave penalty functions. Convergence and other theoretical properties of the LLA algorithm are est…
We propose a two-step pseudo-maximum likelihood procedure for semiparametric single-index regression models where the conditional variance is a known function of the regression and an additional parameter. The Poisson single-index regression with multiplicative unobserved heterogeneity is an example of such models. Our procedure is based on linear exponential densities with nuisance parameter. The pseudo-likelihood criterion we use contains a nonparametric estimate of the index regression and therefore a rule for choosing the smoothing parameter is needed. We propose an automatic and natural r…
We consider estimation in a particular semiparametric regression model for the mean of a counting process with ``panel count'' data. The basic model assumption is that the conditional mean function of the counting process is of the form $E\{\mathbb{N}(t)|Z\}=\exp(β_0^TZ)Λ_0(t)$ where $Z$ is a vector of covariates and $Λ_0$ is the baseline mean function. The ``panel count'' observation scheme involves observation of the counting process $\mathbb{N}$ for an individual at a random number $K$ of random time points; both the number and the locations of these time points may differ across individual…
We provide a semi-parametric analysis for the proportional likelihood ratio model, proposed by Luo & Tsai (2012). We study the tangent spaces for both the parameter of interest and the nuisance parameter, and obtain an explicit expression for the efficient score function. We propose a family of Z-estimators based on the score functions, including an approximated efficient estimator. Using inverse probability weighting, the proposed estimators can also be applied to different missing-data mechanisms, such as right censored data and non-random sampling. A simulation study that illustrates the fi…
This paper develops a general theory on rates of convergence of penalized spline estimators for function estimation when the likelihood functional is concave in candidate functions, where the likelihood is interpreted in a broad sense that includes conditional likelihood, quasi-likelihood, and pseudo-likelihood. The theory allows all feasible combinations of the spline degree, the penalty order, and the smoothness of the unknown functions. According to this theory, the asymptotic behaviors of the penalized spline estimators depends on interplay between the spline knot number and the penalty pa…
The Ising model is a useful tool for studying complex interactions within a system. The estimation of such a model, however, is rather challenging, especially in the presence of high-dimensional parameters. In this work, we propose efficient procedures for learning a sparse Ising model based on a penalized composite conditional likelihood with nonconcave penalties. Nonconcave penalized likelihood estimation has received a lot of attention in recent years. However, such an approach is computationally prohibitive under high-dimensional Ising models. To overcome such difficulties, we extend the m…
In this work we deal with correlated failure time (age at onset) data arising from population-based case-control studies, where case and control probands are selected by population-based sampling and an array of risk factor measures is collected for both cases and controls and their relatives. Parameters of interest are effects of risk factors on the hazard function of failure times and within-family dependencies of failure times after adjusting for the risk factors. Due to the retrospective nature of sampling, a large sample theory for existing methods has not been established. We develop a n…
Predictive recursion is an accurate and computationally efficient algorithm for nonparametric estimation of mixing densities in mixture models. In semiparametric mixture models, however, the algorithm fails to account for any uncertainty in the additional unknown structural parameter. As an alternative to existing profile likelihood methods, we treat predictive recursion as a filter approximation to fitting a fully Bayes model, whereby an approximate marginal likelihood of the structural parameter emerges and can be used for inference. We call this the predictive recursion marginal likelihood.…
We consider a semiparametric mixture of two univariate density functions where one of them is known while the weight and the other function are unknown. Such mixtures have a history of application to the problem of detecting differentially expressed genes under two or more conditions in microarray data. Until now, some additional knowledge about the unknown component (e.g. the fact that it belongs to a location family) has been assumed. As opposed to this approach, we do not assume any additional structure on the unknown density function. For this mixture model, we derive a new sufficient iden…
Weighted likelihood, in which one solves Horvitz-Thompson or inverse probability weighted (IPW) versions of the likelihood equations, offers a simple and robust method for fitting models to two phase stratified samples. We consider semiparametric models for which solution of infinite dimensional estimating equations leads to $\sqrt{N}$ consistent and asymptotically Gaussian estimators of both Euclidean and nonparametric parameters. If the phase two sample is selected via Bernoulli (i.i.d.) sampling with known sampling probabilities, standard estimating equation theory shows that the influence …
This paper deals with a situation when one is interested in the dependence structure of a multidimensional response variable in the presence of a multivariate covariate. It is assumed that the covariate affects only the marginal distributions through regression models while the dependence structure, which is described by a copula, is unaffected. A parametric estimation of the copula function is considered with focus on the maximum pseudo-likelihood method. It is proved that under some appropriate regularity assumptions the estimator calculated from the residuals is asymptotically equivalent to…
Interval-censored multi-state data arise in many studies of chronic diseases, where the health status of a subject can be characterized by a finite number of disease states and the transition between any two states is only known to occur over a broad time interval. We formulate the effects of potentially time-dependent covariates on multi-state processes through semiparametric proportional intensity models with random effects. We adopt nonparametric maximum likelihood estimation (NPMLE) under general interval censoring and develop a stable expectation-maximization (EM) algorithm. We show that …
In this technical report, we consider conditional density estimation with a maximum likelihood approach. Under weak assumptions, we obtain a theoretical bound for a Kullback-Leibler type loss for a single model maximum likelihood estimate. We use a penalized model selection technique to select a best model within a collection. We give a general condition on penalty choice that leads to oracle type inequality for the resulting estimate. This construction is applied to two examples of partition-based conditional density models, models in which the conditional density depends only in a piecewise …
We consider an independence feature screening technique for identifying explanatory variables that locally contribute to the response variable in high-dimensional regression analysis. Without requiring a specific parametric form of the underlying data model, our approach accommodates a wide spectrum of nonparametric and semiparametric model families. To detect the local contributions of explanatory variables, our approach constructs empirical likelihood locally in conjunction with marginal nonparametric regressions. Since our approach actually requires no estimation, it is advantageous in scen…
We would like to take this opportunity to thank the discussants for their thoughtful comments and encouragements on our work [arXiv:0808.1012]. The discussants raised a number of issues from theoretical as well as computational perspectives. Our rejoinder will try to provide some insights into these issues and address specific questions asked by the discussants.
This paper is devoted to the off-line multiple change-point detection in a semiparametric framework. The time series is supposed to belong to a large class of models including AR($\infty$), ARCH($\infty$), TARCH($\infty$),... models where the coefficients change at each instant of breaks. The different unknown parameters (number of changes, change dates and parameters of successive models) are estimated using a penalized contrast built on conditional quasi-likelihood. Under Lipshitzian conditions on the model, the consistency of the estimator is proved when the moment order $r$ of the process …
We address the issue of recovering the structure of large sparse directed acyclic graphs from noisy observations of the system. We propose a novel procedure based on a specific formulation of the l1-norm regularized maximum likelihood, which decomposes the graph estimation into two optimization sub-problems: topological structure and node order learning. We provide oracle inequalities for the graph estimator, as well as an algorithm to solve the induced optimization problem, in the form of a convex program embedded in a genetic algorithm. We apply our method to various data sets (including dat…
We describe here a framework for a certain class of multiscale likelihood factorizations wherein, in analogy to a wavelet decomposition of an L^2 function, a given likelihood function has an alternative representation as a product of conditional densities reflecting information in both the data and the parameter vector localized in position and scale. The framework is developed as a set of sufficient conditions for the existence of such factorizations, formulated in analogy to those underlying a standard multiresolution analysis for wavelets, and hence can be viewed as a multiresolution analys…
Generalized empirical likelihood and generalized method of moments are well spread methods of resolution of inverse problems in econometrics. Each method defines a specific semiparametric model for which it is possible to calculate efficiency bounds. By this approach, we provide a new proof of Chamberlain's result on optimal GMM. We also discuss conditions under which GMM estimators remain efficient with approximate moment constraints.
This paper formulates a penalized empirical likelihood (PEL) method for inference on the population mean when the dimension of the observations may grow faster than the sample size. Asymptotic distributions of the PEL ratio statistic is derived under different component-wise dependence structures of the observations, namely, (i) non-Ergodic, (ii) long-range dependence and (iii) short-range dependence. It follows that the limit distribution of the proposed PEL ratio statistic can vary widely depending on the correlation structure, and it is typically different from the usual chi-squared limit o…
We consider partly linear transformation models applied to current status data. The unknown quantities are the transformation function, a linear regression parameter and a nonparametric regression effect. It is shown that the penalized MLE for the regression parameter is asymptotically normal and efficient and converges at the parametric rate, although the penalized MLE for the transformation function and nonparametric regression effect are only $n^{1/3}$ consistent. Inference for the regression parameter based on a block jackknife is investigated. We also study computational issues and demons…
In this study, we consider unsupervised clustering of categorical vectors that can be of different size using mixture. We use likelihood maximization to estimate the parameters of the underlying mixture model and a penalization technique to select the number of mixture components. Regardless of the true distribution that generated the data, we show that an explicit penalty, known up to a multiplicative constant, leads to a non-asymptotic oracle inequality with the Kullback-Leibler divergence on the two sides of the inequality. This theoretical result is illustrated by a document clustering app…
We find upper bounds for the probability of underestimation and overestimation errors in penalized likelihood context tree estimation. The bounds are explicit and applies to processes of not necessarily finite memory. We allow for general penalizing terms and we give conditions over the maximal depth of the estimated trees in order to get strongly consistent estimates. This generalizes previous results obtained in the case of estimation of the order of a Markov chain.
The paper deals with generalized functional regression. The aim is to estimate the influence of covariates on observations, drawn from an exponential distribution. The link considered has a semiparametric expression: if we are interested in a functional influence of some covariates, we authorize others to be modeled linearly. We thus consider a generalized partially linear regression model with unknown regression coefficients and an unknown nonparametric function. We present a maximum penalized likelihood procedure to estimate the components of the model introducing penalty based wavelet estim…
We consider the problem of regularized maximum likelihood estimation for the structure and parameters of a high-dimensional, sparse directed acyclic graphical (DAG) model with Gaussian distribution, or equivalently, of a Gaussian structural equation model. We show that the $\ell_0$-penalized maximum likelihood estimator of a DAG has about the same number of edges as the minimal-edge I-MAP (a DAG with minimal number of edges representing the distribution), and that it converges in Frobenius norm. We allow the number of nodes p to be much larger than sample size n but assume a sparsity condition…
The purpose of this article is to develop a general parametric estimation theory that allows the derivation of the limit distribution of estimators in non-regular models where the true parameter value may lie on the boundary of the parameter space or where even identifiability fails. For that, we propose a more general local approximation of the parameter space (at the true value) than previous studies. This estimation theory is comprehensive in that it can handle penalized estimation as well as quasi-maximum likelihood estimation under such non-regular models. Besides, our results can apply t…
This paper presents a general theoretical framework of penalized quasi-maximum likelihood (PQML) estimation in stationary multiple time series models when the number of parameters possibly diverges. We show the oracle property of the PQML estimator under high-level, but tractable, assumptions, comprising the first half of the paper. Utilizing these results, we propose in the latter half of the paper a method of sparse estimation in high-dimensional vector autoregressive (VAR) models. Finally, the usability of the sparse high-dimensional VAR model is confirmed with a simulation study and an emp…
