In recent years, bootstrap methods have drawn attention for their ability to approximate the laws of "max statistics" in high-dimensional problems. A leading example of such a statistic is the coordinate-wise maximum of a sample average of $n$ random vectors in $\mathbb{R}^p$. Existing results for this statistic show that the bootstrap can work when $n\ll p$, and rates of approximation (in Kolmogorov distance) have been obtained with only logarithmic dependence in $p$. Nevertheless, one of the challenging aspects of this setting is that established rates tend to scale like $n^{-1/6}$ as a function of $n$.
The main purpose of this paper is to demonstrate that improvement in rate is possible when extra model structure is available. Specifically, we show that if the coordinate-wise variances of the observations exhibit decay, then a nearly $n^{-1/2}$ rate can be achieved, independent of $p$. Furthermore, a surprising aspect of this dimension-free rate is that it holds even when the decay is very weak. Lastly, we provide examples showing how these ideas can be applied to inference problems dealing with functional and multinomial data.
For a multinomial distribution, suppose that we have prior knowledge of the sum of the probabilities of some categories. This allows us to construct a submodel in a full (i.e., no-restriction) model. Maximum likelihood estimation (MLE) under this submodel is expected to have better estimation efficiency than MLE under the full model. This article presents the asymptotic expansion of the risk of MLE with respect to Kullback--Leibler divergence for both the full model and submodel. The results reveal that, using the submodel, the reduction of the risk is quite small in some cases. Furthermore, w…
The multinomial and related distributions have long been used to model categorical, count-based data in fields ranging from bioinformatics to natural language processing. Commonly utilized variants include the standard multinomial and the Dirichlet multinomial distributions due to their computational efficiency and straightforward parameter estimation process. However, these distributions make strict assumptions about the mean, variance, and covariance between the categorical features being modeled. If these assumptions are not met by the data, it may result in poor parameter estimates and los…
An $(n,k)$-Poisson Multinomial Distribution (PMD) is the distribution of the sum of $n$ independent random vectors supported on the set ${\cal B}_k=\{e_1,\ldots,e_k\}$ of standard basis vectors in $\mathbb{R}^k$. We show that any $(n,k)$-PMD is ${\rm poly}\left({k\over σ}\right)$-close in total variation distance to the (appropriately discretized) multi-dimensional Gaussian with the same first two moments, removing the dependence on $n$ from the Central Limit Theorem of Valiant and Valiant. Interestingly, our CLT is obtained by bootstrapping the Valiant-Valiant CLT itself through the structura…
In Quantum Non Demolition measurements, the sequence of observations is distributed as a mixture of multinomial random variables. Parameters of the dynamics are naturally encoded into this family of distributions. We show the local asymptotic mixed normality of the underlying statistical model and the consistency of the maximum likelihood estimator. Furthermore, we prove the asymptotic optimality of this estimator as it saturates the usual Cramér Rao bound.
Estimation of a single Bernoulli parameter using pooled sampling is among the oldest problems in the group testing literature. To carry out such estimation, an array of efficient estimators have been introduced covering a wide range of situations routinely encountered in applications. More recently, there has been growing interest in using group testing to simultaneously estimate the joint probabilities of two correlated traits using a multinomial model. Unfortunately, basic estimation results, such as the maximum likelihood estimator (MLE), have not been adequately addressed in the literature…
In this paper we consider Poisson loglinear models with linear constraints (LMLC) on the expected table counts. Multinomial and product multinomial loglinear models can be obtained by considering that some marginal totals (linear constraints on the expected table counts) have been prefixed in a Poisson loglinear model. Therefore with the theory developed in this paper, multinomial and product multinomial loglinear models can be considered as a particular case. To carry out inferences on the parameters in the LMLC an information-theoretic approach is followed from which the classical maximum li…
We consider a problem of clustering a sequence of multinomial observations by way of a model selection criterion. We propose a form of a penalty term for the model selection procedure. Our approach subsumes both the conventional AIC and BIC criteria but also extends the conventional criteria in a way that it can be applicable also to a sequence of sparse multinomial observations, where even within a same cluster, the number of multinomial trials may be different for different observations. In addition, as a preliminary estimation step to maximum likelihood estimation, and more generally, to ma…
This paper investigates the (in)-consistency of various bootstrap methods for making inference on a change-point in time in the Cox model with right censored survival data. A criterion is established for the consistency of any bootstrap method. It is shown that the usual nonparametric bootstrap is inconsistent for the maximum partial likelihood estimation of the change-point. A new model-based bootstrap approach is proposed and its consistency established. Simulation studies are carried out to assess the performance of various bootstrap schemes.
We study shrinkage estimation of the mean parameters of a class of multivariate distributions for which the diagonal entries of the corresponding covariance matrix are certain quadratic functions of the mean parameter. This class of distributions includes the diagonal multivariate natural exponential families. We propose two classes of semi-parametric shrinkage estimators for the mean and construct unbiased estimators of the corresponding risk. We establish the asymptotic consistency and convergence rates for these shrinkage estimators under squared error loss as both $n$, the sample size, and…
We study the problem of maximum likelihood estimation for general patterns of bivariate missing data for normal and multinomial random variables, under the assumption that the data is missing at random (MAR). For normal data, the score equations have nine complex solutions, at least one of which is real and statistically significant. Our computations suggest that the number of real solutions is related to whether or not the MAR assumption is satisfied. In the multinomial case, all solutions to the score equations are real and the number of real solutions grows exponentially in the number of st…
Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models
School of Computing, National University of Singapore, Singapore.（新加坡国立大学计算机学院）；ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems（细胞系统数学分析卓越研究中心）；School of Mathematical Sciences, Queensland University of Technology, Brisbane, Australia.（昆士兰科技大学数学科学学院）；Department of Statistics and Data Science, University of Texas at Austin, Austin, USA（德克萨斯大学奥斯汀分校统计与数据科学系）
The quasi-maximum likelihood estimation is a commonly-used method for estimating GARCH parameters. However, such estimators are sensitive to outliers and their asymptotic normality is proved under the finite fourth moment assumption on the underlying error distribution. In this paper, we propose a novel class of estimators of the GARCH parameters based on ranks, called R-estimators, with the property that they are asymptotic normal under the existence of a more than second moment of the errors and are highly efficient. We also consider the weighted bootstrap approximation of the finite sample …
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of British Columbia（不列颠哥伦比亚大学）
Out of the rich family of generalized linear bandits, perhaps the most well studied ones are logisitc bandits that are used in problems with binary rewards: for instance, when the learner/agent tries to maximize the profit over a user that can select one of two possible outcomes (e.g., `click' vs `no-click'). Despite remarkable recent progress and improved algorithms for logistic bandits, existing works do not address practical situations where the number of outcomes that can be selected by the user is larger than two (e.g., `click', `show me later', `never show again', `no click'). In this pa…
Recent advances in molecular simulations allow the evaluation of previously unattainable observables, such as rate constants for protein folding. However, these calculations are usually computationally expensive and even significant computing resources may result in a small number of independent estimates spread over many orders of magnitude. Such small-sample, high "log-variance" data are not readily amenable to analysis using the standard uncertainty (i.e., "standard error of the mean") because unphysical negative limits of confidence intervals result. Bootstrapping, a natural alternative gu…
It has been proved that direct bootstrapping of the nonparametric maximum likelihood estimator (MLE) of the distribution function in the current status model leads to inconsistent confidence intervals. We show that bootstrapping of functionals of the MLE can however be used to produce valid intervals. To this end, we prove that the bootstrapped MLE converges at the right rate in the $L_p$-distance. We also discuss applications of this result to the current status regression model.
Enjoying Non-linearity in Multinomial Logistic Bandits: A Minimax-Optimal Algorithm
INRIA（法国国家信息与自动化研究所）；École Normale Supérieure（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立高等物理、数学与信息学院）；SDA Bocconi School of Management（博科尼管理学院）
Combinatorial Inference on the Optimal Assortment in Multinomial Logit Models
Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）；Leonard N. Stern School of Business, New York University（纽约大学伦纳德·N·斯特恩商学院）；Duke University（杜克大学）
A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Univ. Grenoble Alpes（格勒诺布尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LJK（约瑟夫·傅里叶实验室）；The University of Queensland（昆士兰大学）
Learning in Position-Aware Multinomial Logit Bandits: From Multiplicative to General Position Effects
Leonard N. Stern School of Business, New York University（纽约大学勒纳商学院）；Qiuzhen College, Tsinghua University（清华大学齐臻学院）；Department of Management Science, School of Management, Fudan University（复旦大学管理学院管理科学系）；Yau Mathematical Sciences Center & Department of Mathematical Sciences, Tsinghua University（清华大学尤数学科学中心及数学科学系）
The MLE (Maximum Likelihood Estimate) for a multinomial model is proportional to the data. We call such estimate an eigenestimate and the relationship of it to the data as the eigenstructure. When the multinomial model is generalized to deal with data arise from incomplete or censored categorical counts, we would naturally look for this eigenstructure between MLE and data. The paper finds the algebraic representation of the eigenstructure (put as Eqn (2.1)), with which the intuition is visualized geometrically (Figures 2.2 and 4.3) and elaborated in a theory (Section 4). The eigenestimate cons…
In this article we introduce a new family of lattice polytopes with rational linear precision. For this purpose, we define a new class of discrete statistical models that we call multinomial staged tree models. We prove that these models have rational maximum likelihood estimators (MLE) and give a criterion for these models to be log-linear. Our main result is then obtained by applying Garcia-Puente and Sottile's theorem that establishes a correspondence between polytopes with rational linear precision and log-linear models with rational MLE. Throughout this article we also study the interplay…
Leonard N. Stern School of Business, New York University（纽约大学伦纳德·N·斯特恩商学院）；Machine Learning Department, Carnegie Mellon University（卡内基梅隆大学机器学习系）；Computer Science Department, Indiana University at Bloomington（印第安纳大学伯明顿分校计算机科学系）
We study the dynamic assortment planning problem, where for each arriving customer, the seller offers an assortment of substitutable products and customer makes the purchase among offered products according to an uncapacitated multinomial logit (MNL) model. Since all the utility parameters of MNL are unknown, the seller needs to simultaneously learn customers' choice behavior and make dynamic decisions on assortments based on the current knowledge. The goal of the seller is to maximize the expected revenue, or equivalently, to minimize the expected regret. Although dynamic assortment planning …
Tel Aviv University（特拉维夫大学）；The Open University of Israel（以色列开放大学）
In this paper we consider high-dimensional multiclass classification by sparse multinomial logistic regression. We propose first a feature selection procedure based on penalized maximum likelihood with a complexity penalty on the model size and derive the nonasymptotic bounds for misclassification excess risk of the resulting classifier. We establish also their tightness by deriving the corresponding minimax lower bounds. In particular, we show that there exist two regimes corresponding to small and large number of classes. The bounds can be reduced under the additional low noise condition. To…
We present new stochastic geometry theorems that give bounds on the probability that $m$ random data classes all contain a point in common in their convex hulls. We apply these stochastic separation theorems to obtain bounds on the probability of existence of maximum likelihood estimators in multinomial logistic regression. We also discuss connections to condition numbers for analysis of steepest descent algorithms in logistic regression and to the computation of centerpoints of data clouds.
The primal problem of multinomial likelihood maximization restricted to a convex closed subset of the probability simplex is studied. Contrary to widely held belief, a solution of this problem may assign a positive mass to an outcome with zero count. Related flaws in the simplified Lagrange and Fenchel dual problems, which arise because the recession directions are ignored, are identified and corrected. A solution of the primal problem can be obtained by the PP (perturbed primal) algorithm, that is, as the limit of a sequence of solutions of perturbed primal problems. The PP algorithm may be i…
We propose a new approach to the problem of high-dimensional multivariate ANOVA via bootstrapping max statistics that involve the differences of sample mean vectors. The proposed method proceeds via the construction of simultaneous confidence regions for the differences of population mean vectors. It is suited to simultaneously test the equality of several pairs of mean vectors of potentially more than two populations. By exploiting the variance decay property that is a natural feature in relevant applications, we are able to provide dimension-free and nearly-parametric convergence rates for G…
We investigate the performance of model based bootstrap methods for constructing point-wise confidence intervals around the survival function with interval censored data. We show that bootstrapping from the nonparametric maximum likelihood estimator of the survival function is inconsistent for both the current status and case 2 interval censoring models. A model based smoothed bootstrap procedure is proposed and shown to be consistent. In addition, simulation studies are conducted to illustrate the (in)-consistency of the bootstrap methods. Our conclusions in the interval censoring model would…
For the estimation of cumulative link models for ordinal data, the bias-reducing adjusted score equations in \citet{firth:93} are obtained, whose solution ensures an estimator with smaller asymptotic bias than the maximum likelihood estimator. Their form suggests a parameter-dependent adjustment of the multinomial counts, which, in turn suggests the solution of the adjusted score equations through iterated maximum likelihood fits on adjusted counts, greatly facilitating implementation. Like the maximum likelihood estimator, the reduced-bias estimator is found to respect the invariance properti…
We develop a mixtures-of-experts (ME) approach to the multiclass classification where the predictors are univariate functions. It consists of a ME model in which both the gating network and the experts network are constructed upon multinomial logistic activation functions with functional inputs. We perform a regularized maximum likelihood estimation in which the coefficient functions enjoy interpretable sparsity constraints on targeted derivatives. We develop an EM-Lasso like algorithm to compute the regularized MLE and evaluate the proposed approach on simulated and real data.
This article studies bootstrap inference for high dimensional weakly dependent time series in a general framework of approximately linear statistics. The following high dimensional applications are covered: (1) uniform confidence band for mean vector; (2) specification testing on the second order property of time series such as white noise testing and bandedness testing of covariance matrix; (3) specification testing on the spectral property of time series. In theory, we first derive a Gaussian approximation result for the maximum of a sum of weakly dependent vectors, where the dimension of th…
Let X, X_1,X_2,... be a sequence of i.i.d. random variables with mean $μ=E X$. Let ${v_1^{(n)},...,v_n^{(n)}}_{n=1}^\infty$ be vectors of non-negative random variables (weights), independent of the data sequence ${X_1,...,X_n}_{n=1}^\infty$, and put $m_n=\sumn v_i^{(n)}$. Consider $ X^{*}_1,..., X^{*}_{m_n}$, $m_n\geq 1$, a bootstrap sample, resulting from re-sampling or stochastically re-weighing a random sample $X_1,...,X_n$, $n\geq 1$. Put $\bar{X}_n= \sumn X_i/n$, the original sample mean, and define $\bar{X^*}_{m_n}=\sumn v_i^{(n)} X_i/m_n$, the bootstrap sample mean. Thus, $\bar{X^*}_{m_…
