Revisiting Concentration of Missing Mass

Concentration of measure bounds for matrix-variate data with missing values

We consider the following data perturbation model, where the covariates incur multiplicative errors. For two $n \times m$ random matrices $U, X$, we denote by $U \circ X$ the Hadamard or Schur product, which is defined as $(U \circ X)_{ij} = (U_{ij}) \cdot (X_{ij})$. In this paper, we study the subgaussian matrix variate model, where we observe the matrix variate data $X$ through a random mask $U$: $$ {\mathcal X} = U \circ X \; \; \; \text{ where} \; \; \;X = B^{1/2} {\mathbb{Z}} A^{1/2}, $$ where ${\mathbb{Z}}$ is a random matrix with independent subgaussian entries, and $U$ is a mask matrix…

Consistent estimation of the missing mass for feature models

Feature models are popular in machine learning and they have been recently used to solve many unsupervised learning problems. In these models every observation is endowed with a finite set of features, usually selected from an infinite collection $(F_{j})_{j\geq 1}$. Every observation can display feature $F_{j}$ with an unknown probability $p_{j}$. A statistical problem inherent to these models is how to estimate, given an initial sample, the conditional expected number of hitherto unseen features that will be displayed in a future observation. This problem is usually referred to as the missin…

Estimating High-dimensional Covariance and Precision Matrices under General Missing Dependence

A sample covariance matrix $\boldsymbol{S}$ of completely observed data is the key statistic in a large variety of multivariate statistical procedures, such as structured covariance/precision matrix estimation, principal component analysis, and testing of equality of mean vectors. However, when the data are partially observed, the sample covariance matrix from the available data is biased and does not provide valid multivariate procedures. To correct the bias, a simple adjustment method called inverse probability weighting (IPW) has been used in previous research, yielding the IPW estimator. T…

基于样本协方差矩阵的完全随机缺失检验

Tests of Missing Completely At Random based on sample covariance matrices

本研究将高维数据的完全随机缺失检验放宽为协方差矩阵集合的相容性检验，定义了基于半定规划的不相容性度量，提出经自助法校准的假设检验方法，证明其在部分场景下的极小极大最优性，并验证了重尾数据下的有效性。

高阶缺失质量的最优估计，以及稀有类型匹配问题

Optimal estimation of high-order missing masses, and the rare-type match problem

该研究针对高阶缺失质量估计问题，提出非参数估计量与非渐近置信区间，证明极小极大估计需更强尾部假设，所得估计为最优极小极大估计，并为法医统计学中的似然比估计提供理论支撑。