We study approaches to robust model-based design of experiments in the context of maximum-likelihood estimation. These approaches provide robustification of model-based methodologies for the design of optimal experiments by accounting for the effect of the parametric uncertainty. We study the problem of robust optimal design of experiments in the framework of nonlinear least-squares parameter estimation using linearized confidence regions. We investigate several well-known robustification frameworks in this respect and propose a novel methodology based on multi-stage robust optimization. The proposed methodology aims at problems, where the experiments are designed sequentially with a possibility of re-estimation in-between the experiments. The multi-stage formalism aids in identifying experiments that are better conducted in the early phase of experimentation, where parameter knowledge is poor. We demonstrate the findings and effectiveness of the proposed methodology using four case studies of varying complexity.
Hyogo University of Health Sciences（兵库医疗大学）；Chiba University（千叶大学）
This study proposes multivariate kernel density estimation by stagewise minimization algorithm based on $U$-divergence and a simple dictionary. The dictionary consists of an appropriate scalar bandwidth matrix and a part of the original data. The resulting estimator brings us data-adaptive weighting parameters and bandwidth matrices, and realizes a sparse representation of kernel density estimation. We develop the non-asymptotic error bound of estimator obtained via the proposed stagewise minimization algorithm. It is confirmed from simulation studies that the proposed estimator performs compe…
Car-sharing problem is a popular research field in sharing economy. In this paper, we investigate the car-sharing re-balancing problem under uncertain demands. An innovative framework that integrates a non-parametric approach - kernel density estimation (KDE) and a two-stage stochastic programming (SP) model are proposed. Specifically, the probability distributions are derived from New York taxi trip data sets by KDE, which is used as the input uncertain parameters for SP. Additionally, the car-sharing problem is formulated as a two-stage SP model which aims to maximize the overall profit. Mea…
Applications of structural equation models (SEMs) are often restricted to linear associations between variables. Maximum likelihood (ML) estimation in non-linear models may be complex and require numerical integration. Furthermore, ML inference is sensitive to distributional assumptions. In this paper, we introduce a simple two-stage estimation technique for estimation of non-linear associations between latent variables. Here both steps are based on fitting linear SEMs: first a linear model is fitted to data on the latent predictor and terms describing the non-linear effect are predicted by th…
Difficulty in identifying cancer stage in health care claims data has limited oncology quality of care and health outcomes research. We fit prediction algorithms for classifying lung cancer stage into three classes (stages I/II, stage III, and stage IV) using claims data, and then demonstrate a method for incorporating the classification uncertainty in outcomes estimation. Leveraging set-valued classification and split conformal inference, we show how a fixed algorithm developed in one cohort of data may be deployed in another, while rigorously accounting for uncertainty from the initial class…
We consider a re-sampling scheme for estimation of the population parameters in the mixed effects nonlinear regression models of the type use for example in clinical pharmacokinetics, say. We provide an estimation procedure which {\it recycles}, via random weighting, the relevant two-stage parameters estimates to construct consistent estimates of the sampling distribution of the various estimates. We establish the asymptotic consistency and asymptotic normality of the resampled estimates and demonstrate the applicability of the {\it recycling} approach in a small simulation study and via examp…
Future University Hakodate（函馆未来大学）；National Graduate Institute for Policy Studies（政策研究大学院大学）；RIKEN AIP（理化学研究所先进智能项目中心）
Contrastive learning is a recent promising approach in unsupervised representation learning where a feature representation of data is learned by solving a pseudo classification problem from unlabelled data. However, it is not straightforward to understand what representation contrastive learning yields. In addition, contrastive learning is often based on the maximum likelihood estimation, which tends to be vulnerable to the contamination by outliers. To promote the understanding to contrastive learning, this paper first theoretically shows a connection to maximization of mutual information (MI…
A function based nonlinear least squares estimation (FNLSE) method is proposed and investigated in parameter estimation of Jelinski-Moranda software reliability model. FNLSE extends the potential fitting functions of traditional least squares estimation (LSE), and takes the logarithm transformed nonlinear least squares estimation (LogLSE) as a special case. A novel power transformation function based nonlinear least squares estimation (powLSE) is proposed and applied to the parameter estimation of Jelinski-Moranda model. Solved with Newton-Raphson method, Both LogLSE and powLSE of Jelinski-Mor…
We consider a resampling scheme for parameters estimates in nonlinear regression models. We provide an estimation procedure which recycles, via random weighting, the relevant parameters estimates to construct consistent estimates of the sampling distribution of the various estimates. We establish the asymptotic normality of the resampled estimates and demonstrate the applicability of the recycling approach in a small simulation study and via example.
This paper considers a nonlinear quantile model with change-points. The quantile estimation method, which as a particular case includes median model, is more robust with respect to other traditional methods when model errors contain outliers. Under relatively weak assumptions, the convergence rate and asymptotic distribution of change-point and of regression parameter estimators are obtained. Numerical study by Monte Carlo simulations shows the performance of the proposed method for nonlinear model with change-points.
We present a practical approach for computing the sandwich variance estimator in two-stage regression model settings. As a motivating example for two-stage regression, we consider regression calibration, a popular approach for addressing covariate measurement error. The sandwich variance approach has been rarely applied in regression calibration, despite that it requires less computation time than popular resampling approaches for variance estimation, specifically the bootstrap. This is likely due to requiring specialized statistical coding. In practice, a simple bootstrap approach with Wald c…
We consider the problem of threshold estimation for autoregressive time series with a "space switching" in the situation, when the regression is nonlinear and the innovations have a smooth, possibly non Gaussian, probability density. Assuming that the unknown threshold parameter is sampled from a continuous positive density, we find the asymptotic distribution of the Bayes estimator. As usually in the singular estimation problems, the sequence of Bayes estimators is found to be asymptotically efficient, attaining the minimax risk lower bound.
We study the estimation of $β$ for the nonlinear model $y = f(X\sp{\top}β) + ε$ when $f$ is a nonlinear transformation that is known, $β$ has sparse nonzero coordinates, and the number of observations can be much smaller than that of parameters ($n\ll p$). We show that in order to bound the $L_2$ error of the $L_0$ regularized estimator $\hatβ$, i.e., $\|\hatβ- β\|_2$, it is sufficient to establish two conditions. Based on this, we obtain bounds of the $L_2$ error for (1) $L_0$ regularized maximum likelihood estimation (MLE) for exponential linear models and (2) $L_0$ regularized least square …
Truncated conditional expectation functions are objects of interest in a wide range of economic applications, including income inequality measurement, financial risk management, and impact evaluation. They typically involve truncating the outcome variable above or below certain quantiles of its conditional distribution. In this paper, based on local linear methods, a novel, two-stage, nonparametric estimator of such functions is proposed. In this estimation problem, the conditional quantile function is a nuisance parameter that has to be estimated in the first stage. The proposed estimator is …
This paper presents a novel algorithm for efficient online estimation of the filter derivatives in general hidden Markov models. The algorithm, which has a linear computational complexity and very limited memory requirements, is furnished with a number of convergence results, including a central limit theorem with an asymptotic variance that can be shown to be uniformly bounded in time. Using the proposed filter derivative estimator we design a recursive maximum likelihood algorithm updating the parameters according the gradient of the one-step predictor log-likelihood. The efficiency of this …
A two-stage batch estimation algorithm for solving a class of nonlinear, static parameter estimation problems that appear in aerospace engineering applications is proposed. It is shown how these problems can be recast into a form suitable for the proposed two-stage estimation process. In the first stage, linear least squares is used to obtain a subset of the unknown parameters (set 1), while a residual sampling procedure is used for selecting initial values for the rest of the parameters (set 2). In the second stage, depending on the uniqueness of the local minimum, either only the parameters …
We propose two-stage and sequential procedures to estimate the unknown parameter N of a binomial distribution with unknown parameter p, when we reinforce data with an independent sample of a negative-binomial experiment having the same p.
The two-stage least-squares (2SLS) estimator is known to be biased when its first-stage fit is poor. I show that better first-stage prediction can alleviate this bias. In a two-stage linear regression model with Normal noise, I consider shrinkage in the estimation of the first-stage instrumental variable coefficients. For at least four instrumental variables and a single endogenous regressor, I establish that the standard 2SLS estimator is dominated with respect to bias. The dominating IV estimator applies James-Stein type shrinkage in a first-stage high-dimensional Normal-means problem follow…
Many problems on signal processing reduce to nonparametric function estimation. We propose a new methodology, piecewise convex fitting (PCF), and give a two-stage adaptive estimate. In the first stage, the number and location of the change points is estimated using strong smoothing. In the second stage, a constrained smoothing spline fit is performed with the smoothing level chosen to minimize the MSE. The imposed constraint is that a single change point occurs in a region about each empirical change point of the first-stage estimate. This constraint is equivalent to requiring that the third d…
This paper develops maximum score estimation of preference parameters in the binary choice model under uncertainty in which the decision rule is affected by conditional expectations. The preference parameters are estimated in two stages: we estimate conditional expectations nonparametrically in the first stage and then the preference parameters in the second stage based on Manski (1975, 1985)'s maximum score estimator using the choice data and first stage estimates. The paper establishes consistency and derives rate of convergence of the two-stage maximum score estimator. Moreover, the paper a…
This paper considers the state estimation problem for nonlinear dynamic systems with unknown but bounded noises. Set membership filter (SMF) is a popular algorithm to solve this problem. In the set membership setting, we investigate the filter problem where the state estimation requires to be constrained by a linear or nonlinear equality. We propose a consensus alternating direction method of multipliers (ADMM) based SMF algorithm for nonlinear dynamic systems. To deal with the difficulty of nonlinearity, instead of linearizing the nonlinear system, a semi-infinite programming (SIP) approach i…
Multi-stage (designed) procedures, obtained by splitting the sampling budget suitably across stages, and designing the sampling at a particular stage based on information about the parameter obtained from previous stages, are often advantageous from the perspective of precise inference. We develop a generic framework for M-estimation in a multistage setting and apply empirical process techniques to develop limit theorems that describe the large sample behavior of the resulting M-estimates. Applications to change-point estimation, inverse isotonic regression, classification and mode estimation …
The multivariate probit is popular for modeling correlated binary data, with an attractive balance of flexibility and simplicity. However, considerable challenges remain in computation and in devising a clear statistical framework. Interest in the multivariate probit has increased in recent years. Current applications include genomics and precision medicine, where simultaneous modeling of multiple traits may be of interest, and computational efficiency is an important consideration. We propose a fast method for multivariate probit estimation via a two-stage composite likelihood. We explore com…
We consider conditional estimation in two-stage sample size adjustable designs and the following bias. More specifically, we consider a design which permits raising the sample size when interim results look rather promising, and, which keeps the originally planned sample size when results look very promising. The estimation procedures reported comprise the unconditional maximum likelihood, the conditionally unbiased Rao-Blackwell estimator, the conditional median unbiased estimator, and the conditional maximum likelihood with and without bias correction. We compare these estimators based on an…
Columbia University（哥伦比亚大学）；Pontifical Catholic University of Rio de Janeiro (PUC-Rio)（里约热内卢天主教大学）；University of California, Irvine（加州大学尔湾分校）
In this paper, we introduce a new machine learning (ML) model for nonlinear regression called the Boosted Smooth Transition Regression Trees (BooST), which is a combination of boosting algorithms with smooth transition regression trees. The main advantage of the BooST model is the estimation of the derivatives (partial effects) of very general nonlinear models. Therefore, the model can provide more interpretation about the mapping between the covariates and the dependent variable than other tree-based models, such as Random Forests. We present several examples with both simulated and real data…
In this paper, an approach to estimating a nonlinear deterministic model is presented. We introduce a stochastic model with extremely small variances so that the deterministic and stochastic models are essentially indistinguishable from each other. This point is explained in the paper. The estimation is then carried out using stochastic optimisation based on Markov chain Monte Carlo (MCMC) methods.
In this paper we apply a two-stage sequential design to item calibration problems under a three-parameter logistic model assumption. The measurement errors of the estimates of the latent trait levels of examinees are considered in our procedure. Moreover, a sequential procedure is employed to guarantee that the estimates of the parameters reach a prescribed accuracy criterion when the iteration is stopped, which fully takes the advantage of sequential design. Statistical properties of both the item parameter estimates and the sequential procedure are discussed. We compare the performance of th…
There has been recent growth in small area estimation due to the need for more precise estimation of small geographic areas, which has led to groups such as the U.S. Census Bureau, Google, and the RAND corporation utilizing small area estimation procedures. We develop novel two-stage benchmarking methodology using a single weighted squared error loss function that combines the loss at the unit level and the area level without any specific distributional assumptions. We consider this loss while benchmarking the weighted means at each level or both the weighted means and weighted variability at …
We study the problem of estimating the mode and maximum of an unknown regression function in the presence of noise. We adopt the Bayesian approach by using tensor-product B-splines and endowing the coefficients with Gaussian priors. In the usual fixed-in-advanced sampling plan, we establish posterior contraction rates for mode and maximum and show that they coincide with the minimax rates for this problem. To quantify estimation uncertainty, we construct credible sets for these two quantities that have high coverage probabilities with optimal sizes. If one is allowed to collect data sequential…
State-space models (SSMs) are a popular tool for modeling animal abundances. Inference difficulties for simple linear SSMs are well known, particularly in relation to simultaneous estimation of process and observation variances. Several remedies to overcome estimation problems have been studied for relatively simple SSMs, but whether these challenges and proposed remedies apply for nonlinear stage-structured SSMs, an important class of ecological models, is less well understood. Here we identify improvements for inference about nonlinear stage-structured SSMs fit with biased sequential life st…
We propose a new 2-stage procedure that relies on the elastic net penalty to estimate a network based on partial correlations when data are heavy-tailed. The new estimator allows to consider the lasso penalty as a special case. Using Monte Carlo simulations, we test the performance on several underlying network structures and four different multivariate distributions: Gaussian, t-Student with 3 and 20 degrees of freedom and contaminated Gaussian. Simulation analysis shows that the 2-stage estimator performs best for heavy-tailed data and it is also robust to distribution misspecification, both…
Nonlinear (systems of) ordinary differential equations (ODEs) are common tools in the analysis of complex one-dimensional dynamic systems. In this paper we propose a smoothing approach regularized by a quasilinearized ODE-based penalty in order to approximate the state functions and estimate the parameters defining nonlinear differential systems from noisy data. Within the quasilinearized spline based framework, the estimation process reduces to a conditionally linear problem for the optimization of the spline coefficients. Furthermore, standard ODE compliance parameter(s) selection criteria a…
Numerous variable selection methods rely on a two-stage procedure, where a sparsity-inducing penalty is used in the first stage to predict the support, which is then conveyed to the second stage for estimation or inference purposes. In this framework, the first stage screens variables to find a set of possibly relevant variables and the second stage operates on this set of candidate variables, to improve estimation accuracy or to assess the uncertainty associated to the selection of variables. We advocate that more information can be conveyed from the first stage to the second one: we use the …
