A unified approach for covariance matrix estimation under Stein loss

矩阵正态分布的Stein方法

Stein's method for the matrix normal distribution

本文首次系统发展矩阵分布的Stein方法，通过矩阵Ornstein-Uhlenbeck扩散建立Stein恒等式，给出Stein方程解的半群表示及正则性估计，并应用于矩阵中心极限定理、矩阵T分布近似及协方差估计。

Likelihood Geometry of Correlation Models

Correlation matrices are standardized covariance matrices. They form an affine space of symmetric matrices defined by setting the diagonal entries to one. We study the geometry of maximum likelihood estimation for this model and linear submodels that encode additional symmetries. We also consider the problem of minimizing two closely related functions of the covariance matrix: the Stein's loss and the symmetrized Stein's loss. Unlike the Gaussian log-likelihood these two functions are convex and hence admit a unique positive definite optimum. Some of our results hold for general affine covaria…

Covariance estimation with nonnegative partial correlations

We study the problem of high-dimensional covariance estimation under the constraint that the partial correlations are nonnegative. The sign constraints dramatically simplify estimation: the Gaussian maximum likelihood estimator is well defined with only two observations regardless of the number of variables. We analyze its performance in the setting where the dimension may be much larger than the sample size. We establish that the estimator is both high-dimensionally consistent and minimax optimal in the symmetrized Stein loss. We also prove a negative result which shows that the sign-constrai…

Coherence of high-dimensional random matrices in a Gaussian case : application of the Chen-Stein method

This paper studies the $τ$-coherence of a (n x p)-observation matrix in a Gaussian framework. The $τ$-coherence is defined as the largest magnitude outside a diagonal bandwith of size $τ$ of the empirical correlation coefficients associated to our observations. Using the Chen-Stein method we derive the limiting law of the normalized coherence and show the convergence towards a Gumbel distribution. We generalize here the results of Cai and Jiang [CJ11a]. We assume that the covariance matrix of the model is bandwise. Moreover, we provide numerical considerations highlighting issues from the high…

关于协方差矩阵的正交等变估计量

On the orthogonally equivariant estimators of a covariance matrix

针对高维场景下的总体协方差矩阵估计问题，本文利用Marčenko-Pastur方程推导的显式等式构建新型正交等变估计量，证明其特征值估计的一致性及归一化Stein损失下的最优性。

Optimal Uncertainty Size in Distributionally Robust Inverse Covariance Estimation

In a recent paper, Nguyen, Kuhn, and Esfahani (2018) built a distributionally robust estimator for the precision matrix of the Gaussian distribution. The distributional uncertainty size is a key ingredient in the construction of this estimator. We develop a statistical theory which shows how to optimally choose the uncertainty size to minimize the associated Stein loss. Surprisingly, rather than the expected canonical square-root scaling rate, the optimal uncertainty size scales linearly with the sample size.

多元复正态分布均值矩阵的收缩估计

Shrinkage estimation of a mean matrix of a multivariate complex normal distribution

在不变损失函数下，考虑了协方差阵未知的多元复正态分布均值阵的估计问题。利用Stein恒等式、Stein-Haff恒等式和特征值微积分的复变形式，得到了一类不变估计量风险的无偏估计公式，并由此构造了几个极小极大收缩估计.

High-Dimensional Covariance Shrinkage for Signal Detection

In this paper, we consider the problem of determining the presence of a given signal in a high-dimensional observation with unknown covariance matrix by using an adaptive matched filter. Traditionally such filters are formed from the sample covariance matrix of some given training data, but, as is well-known, the performance of such filters is poor when the number of training data $n$ is not much larger than the data dimension $p$. We thus seek a covariance estimator to replace sample covariance. To account for the fact that $n$ and $p$ may be of comparable size, we adopt the "large-dimensiona…

Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model

We show that in a common high-dimensional covariance model, the choice of loss function has a profound effect on optimal estimation. In an asymptotic framework based on the Spiked Covariance model and use of orthogonally invariant estimators, we show that optimal estimation of the population covariance matrix boils down to design of an optimal shrinker $η$ that acts elementwise on the sample eigenvalues. Indeed, to each loss function there corresponds a unique admissible eigenvalue shrinker $η^*$ dominating all other shrinkers. The shape of the optimal shrinker is determined by the choice of l…

核Stein差异的极小极大估计：迹与希尔伯特-施密特尺度

Minimax Estimation of Kernel Stein Discrepancy: Trace versus Hilbert-Schmidt Scales

研究从n个独立观测估计核Stein差异，确定控制极小极大风险的尖锐谱常数，正部平方根U统计量达最优尺度，标准插件V统计量次优。

A sharp boundary for SURE-based admissibility for the Normal means problem under unknown scale

We consider quasi-admissibility/inadmissibility of Stein-type shrinkage estimators of the mean of a multivariate normal distribution with covariance matrix an unknown multiple of the identity. Quasi-admissibility/inadmissibility is defined in terms of non-existence/existence of a solution to a differential inequality based on Stein's unbiased risk estimate (SURE). We find a sharp boundary between quasi-admissible and quasi-inadmissible estimators related to the optimal James-Stein estimator. We also find a class of priors related to the Strawderman class in the known variance case where the bo…

关于改进James-Stein估计量的几点注记

Some notes on improving upon the James-Stein estimator

本文考虑在平方误差损失和下多元正态均值向量向量的估计。本文提出了一类新的光滑估计，其参数α支配James-Stein估计.α=1的估计对应于关于调和先验的广义贝叶斯估计。当α趋于无穷大时，估计量收敛于James-Stein正部估计量。因此，我们的估计类是可容许估计（α=1）和不可容许估计（α= \infty）之间的桥梁。虽然估计量具有拟容许性，这是一个比容许性弱的最优性，但对于α>1的估计量是否容许的问题仍然是一个悬而未决的问题。

Covariance loss, Szemeredi regularity, and differential privacy

We show how randomized rounding based on Grothendieck's identity can be used to prove a nearly tight bound on the covariance loss--the amount of covariance that is lost by taking conditional expectation. This result yields a new type of weak Szemeredi regularity lemma for positive semidefinite matrices and kernels. Moreover, it can be used to construct differentially private synthetic data.

基于正则化Cholesky因子的奇异情形协方差矩阵估计

Covariance matrix estimation in the singular case using regularized Cholesky factor

针对样本量不足导致样本协方差矩阵奇异的估计问题，提出基于增强部分Cholesky分解的正则化方法，推导有限样本最优与Oracle估计器并补全分解，数值实验验证了方法性能并分析了条件数与谱形状的影响。

重新思考均方误差：信息、广义估计与詹姆斯 - 斯坦悖论

Rethinking Mean Square Error: Information, Generalized Estimation, and the James-Stein Paradox

研究詹姆斯 - 斯坦悖论，将均方误差与Λ - 信息作评估标准比较，基于点估计量与广义估计量等区别，通过基本引理等阐述，解决悖论，明确最大似然估计充分有效，均方误差在特定损失下有其领域。

可分解高斯图模型的协方差估计

Covariance estimation in decomposable Gaussian graphical models

图模型是一种使用图表示和利用分布中先验条件独立结构的框架。在高斯情况下，这些模型与逆协方差（浓度）矩阵的稀疏性直接相关，并且允许以较低的计算复杂性改进协方差估计。我们认为浓度估计的均方误差（MSE）为目标，在一个特殊类型的模型称为分解。这种模式包括，例如，众所周知的带状结构和其他情况下遇到的实践。我们的第一个贡献是推导和分析的最小方差无偏估计（MVUE）在可分解的图形模型。我们提供了一个简单的封闭形式的MVUE的解决方案，并比较它与经典的最大似然估计（MLE）的性能和复杂性。接下来，我们将著名的Stein无偏风险估计（SURE）扩展到图形模型。使用SURE，我们证明了MVUE的MSE总是小于或等于偏置MLE的MSE，并且MVUE本身由其他方法主导。此外，我们建议使用的SURE作为一个建设性的机制，获得新的协方差估计。类似于经典的极大似然估计，我们提出的所有估计都有简单的封闭形式的解决方案，但在MSE显着减少的结果。

Pretest and Stein-Type Estimations in Quantile Regression Model

In this study, we consider preliminary test and shrinkage estimation strategies for quantile regression models. In classical Least Squares Estimation (LSE) method, the relationship between the explanatory and explained variables in the coordinate plane is estimated with a mean regression line. In order to use LSE, there are three main assumptions on the error terms showing white noise process of the regression model, also known as Gauss-Markov Assumptions, must be met: (1) The error terms have zero mean, (2) The variance of the error terms is constant and (3) The covariance between the errors …

多元正态分布方差-协方差阵特征值的可容许估计--详细证明--

Admissible Estimator of the Eigenvalues of Variance-Covariance Matrix for Multivariate Normal Distributions--Detailed Proof--

在尺度不变平方误差损失下，给出了多元正态分布的方差-协方差阵特征值的一个可容许估计。

Covariance matrix estimation under data-based loss

In this paper, we consider the problem of estimating the $p\times p$ scale matrix $Σ$ of a multivariate linear regression model $Y=X\,β+ \mathcal{E}\,$ when the distribution of the observed matrix $Y$ belongs to a large class of elliptically symmetric distributions. After deriving the canonical form $(Z^T U^T)^T$ of this model, any estimator $\hat{ Σ}$ of $Σ$ is assessed through the data-based loss tr$(S^{+}Σ\, (Σ^{-1}\hatΣ - I_p)^2 )\,$ where $S=U^T U$ is the sample covariance matrix and $S^{+}$ is its Moore-Penrose inverse. We provide alternative estimators to the usual estimators $a\,S$, wh…

Optimal Covariance Estimation for Condition Number Loss in the Spiked Model

We study estimation of the covariance matrix under relative condition number loss $κ(Σ^{-1/2} \hatΣ Σ^{-1/2})$, where $κ(Δ)$ is the condition number of matrix $Δ$, and $\hatΣ$ and $Σ$ are the estimated and theoretical covariance matrices. Optimality in $κ$-loss provides optimal guarantees in two stylized applications: Multi-User Covariance Estimation and Multi-Task Linear Discriminant Analysis. We assume the so-called spiked covariance model for $Σ$, and exploit recent advances in understanding that model, to derive a nonlinear shrinker which is asymptotically optimal among orthogonally-equiva…

关于一般损失函数下一般二元对称分布的序约束位置参数同时估计的一个注记

A Note On Simultaneous Estimation of Order Restricted Location Parameters of a General Bivariate Symmetric Distribution Under a General Loss Function

本文研究一般损失函数下二元对称分布中受序约束位置参数的同时估计，利用 Stein 与 Kubokawa（IERD）技术构造优于任意位置等变估计量的稳健改进估计量，并通过模拟和实际数据验证。

Bayesian Estimation of Sparse Spiked Covariance Matrices in High Dimensions

We propose a Bayesian methodology for estimating spiked covariance matrices with jointly sparse structure in high dimensions. The spiked covariance matrix is reparametrized in terms of the latent factor model, where the loading matrix is equipped with a novel matrix spike-and-slab LASSO prior, which is a continuous shrinkage prior for modeling jointly sparse matrices. We establish the rate-optimal posterior contraction for the covariance matrix with respect to the operator norm as well as that for the principal subspace with respect to the projection operator norm loss. We also study the poste…

Local inversion-free estimation of spatial Gaussian processes

Maximizing the likelihood has been widely used for estimating the unknown covariance parameters of spatial Gaussian processes. However, evaluating and optimizing the likelihood function can be computationally intractable, particularly for large number of (possibly) irregularly spaced observations, due to the need to handle the inverse of ill-conditioned and large covariance matrices. Extending the "inversion-free" method of Anitescu, Chen and Stein \cite{anitescu2017inversion}, we investigate a broad class of covariance parameter estimators based on inversion-free surrogate losses and block di…

SURE Information Criteria for Large Covariance Matrix Estimation and Their Asymptotic Properties

Consider $n$ independent and identically distributed $p$-dimensional Gaussian random vectors with covariance matrix $Σ.$ The problem of estimating $Σ$ when $p$ is much larger than $n$ has received a lot of attention in recent years. Yet little is known about the information criterion for covariance matrix estimation. How to properly define such a criterion and what are the statistical properties? We attempt to answer these questions in the present paper by focusing on the estimation of bandable covariance matrices when $p>n$ but $\log(p)=o(n)$. Motivated by the deep connection between Stein's …

泛函线性模型与单指标模型：基于高斯Stein恒等式的统一方法

Functional linear and single-index models: A unified approach via Gaussian Stein identity

本文基于高斯Stein恒等式，为泛函线性与单指标模型提供了统一的估计框架，证明了RKHS最小二乘估计量可同时用于单指标模型且无需指定链接函数，并刻画了收敛速度，放宽了算子假设并允许指标误设。

带凸正则项的Stein无偏风险估计的易处理评估方法

Tractable Evaluation of Stein's Unbiased Risk Estimate with Convex Regularizers

针对带凸正则项的估计器中Stein无偏风险估计现有通用方法无法适配大规模问题的局限，提出可处理宽类估计器且适用于大规模问题的SURE评估方法。

最小最大与自适应协方差矩阵估计在差分隐私下的研究

Minimax and Adaptive Covariance Matrix Estimation under Differential Privacy

本文研究在差分隐私约束下高维带状协方差矩阵的最小最大与自适应估计，提出一种新的差分隐私分块三对角估计器，达到运算规范和弗罗贝尼乌斯范数下的最优收敛速率，并引入自适应估计器在不预知衰减参数的情况下达到最优速率。

Robust Generalised Bayesian Inference for Intractable Likelihoods

Generalised Bayesian inference updates prior beliefs using a loss function, rather than a likelihood, and can therefore be used to confer robustness against possible mis-specification of the likelihood. Here we consider generalised Bayesian inference with a Stein discrepancy as a loss function, motivated by applications in which the likelihood contains an intractable normalisation constant. In this context, the Stein discrepancy circumvents evaluation of the normalisation constant and produces generalised posteriors that are either closed form or accessible using standard Markov chain Monte Ca…

Covariance's Loss is Privacy's Gain: Computationally Efficient, Private and Accurate Synthetic Data

The protection of private information is of vital importance in data-driven research, business, and government. The conflict between privacy and utility has triggered intensive research in the computer science and statistics communities, who have developed a variety of methods for privacy-preserving data release. Among the main concepts that have emerged are anonymity and differential privacy. Today, another solution is gaining traction, synthetic data. However, the road to privacy is paved with NP-hard problems. In this paper we focus on the NP-hard challenge to develop a synthetic data gener…

基于矩阵损失函数的收缩估计

Shrinkage estimation with a matrix loss function

考虑估计具有常方差的独立正态分布观测的n × p矩阵的均值的n × p矩阵，其中估计量的性能使用p × p矩阵二次误差损失函数来判断。一个矩阵版本的James-Stein估计提出，取决于一个调谐常数。当n大于或等于3时，对于某些调谐常数的选择，它将主导通常的极大似然估计。该结果也扩展到其他收缩估计量和设置。

Parameter estimations for the Gaussian process with drift at discrete observation

This paper first strictly proved that the growth of the second moment of a large class of Gaussian processes is not greater than power function and the covariance matrix is strictly positive definite. Under these two conditions, the maximum likelihood estimators of the mean and variance of such classes of drift Gaussian process have strong consistency under broader growth of t_n. At the same time, the asymptotic normality of binary random vectors and the Berry-Esséen bound of estimators are obtained by using the Stein's method via Malliavian calculus.

The Local Ledoit-Peche Law

Ledoit and Peche proved convergence of certain functions of a random covariance matrix's resolvent; we refer to this as the Ledoit-Peche law. One important application of their result is shrinkage covariance estimation with respect to so-called Minimum Variance (MV) loss, discussed in the work of Ledoit and Wolf. We provide an essentially optimal rate of convergence and hypothesize it to be the smallest possible rate of excess MV loss within the shrinkage class.

A Well-Conditioned and Sparse Estimation of Covariance and Inverse Covariance Matrices Using a Joint Penalty

We develop a method for estimating well-conditioned and sparse covariance and inverse covariance matrices from a sample of vectors drawn from a sub-gaussian distribution in high dimensional setting. The proposed estimators are obtained by minimizing the quadratic loss function and joint penalty of `1 norm and variance of its eigenvalues. In contrast to some of the existing methods of covariance and inverse covariance matrix estimation, where often the interest is to estimate a sparse matrix, the proposed method is flexible in estimating both a sparse and well-conditioned covariance matrix simu…

Inference on the eigenvalues of the covariance matrix of a multivariate normal distribution--geometrical view--

We consider an inference on the eigenvalues of the covariance matrix of a multivariate normal distribution. The family of multivariate normal distributions with a fixed mean is seen as a Riemannian manifold with Fisher information metric. Two submanifolds naturally arises; one is the submanifold given by fixed eigenvectors of the covariance matrix, the other is the one given by fixed eigenvalues. We analyze the geometrical structures of these manifolds such as metric, embedding curvature under $e$-connection or $m$-connection. Based on these results, we study 1) the bias of the sample eigenval…

尖峰协方差模型中的噪声估计

Noise Estimation in the Spiked Covariance Model

研究了Frobenius损失下高维加样协方差阵的估计问题和加样PCA中噪声估计的并行问题。我们提出了一个估计的噪声参数最小化的无偏估计的不变Frobenius风险使用变分法。由此产生的估计，使用随机矩阵理论，是强一致的，基本上是渐近正常和极小极大的噪声估计问题。我们应用该构造构造了一个特征值一致的鲁棒尖峰协方差阵估计。

稀疏协方差阵估计的最优收敛速度

Optimal rates of convergence for sparse covariance matrix estimation

本文考虑稀疏协方差阵的估计，在一定的矩阵算子范数和Bregman发散损失下，建立了最优的收敛速率。一个主要的重点是推导出一个率夏普极小极大下界。该问题表现出新的功能，显着不同于那些发生在传统的非参数函数估计问题。标准技术不能产生良好的结果，因此需要新的工具。我们首先开发了一个下界技术，特别适合于治疗“双向”的问题，如估计稀疏的协方差阵。这个结果可以看作是Le Cam方法和Assouad引理的一个推广。这种下界技术是独立的利益，并可用于其他矩阵估计问题。然后，我们建立了一个率夏普minimax下界估计稀疏协方差阵谱范数下，通过应用一般的下界技术。一个阈值估计，以达到最佳的收敛速率下的谱范数。然后将结果推广到一般矩阵$\ell_w$算子范数的$1\le w\le \infty$。此外，在一类Bregman发散损失下，我们给出了稀疏协方差阵估计的Minimax收敛速率的统一结果。

Optimal Bayesian Minimax Rates for Unconstrained Large Covariance Matrices

We obtain the optimal Bayesian minimax rate for the unconstrained large covariance matrix of multivariate normal sample with mean zero, when both the sample size, n, and the dimension, p, of the covariance matrix tend to infinity. Traditionally the posterior convergence rate is used to compare the frequentist asymptotic performance of priors, but defining the optimality with it is elusive. We propose a new decision theoretic framework for prior selection and define Bayesian minimax rate. Under the proposed framework, we obtain the optimal Bayesian minimax rate for the spectral norm for all rat…

Estimation under matrix quadratic loss and matrix superharmonicity

We investigate estimation of a normal mean matrix under the matrix quadratic loss. Improved estimation under the matrix quadratic loss implies improved estimation of any linear combination of the columns. First, an unbiased estimate of risk is derived and the Efron--Morris estimator is shown to be minimax. Next, a notion of \textit{matrix superharmonicity} for matrix-variate functions is introduced and shown to have analogous properties with usual superharmonic functions, which may be of independent interest. Then, we show that the generalized Bayes estimator with respect to a matrix superharm…

矩阵范数收缩估计量与先验

Matrix norm shrinkage estimators and priors

该研究针对Frobenius损失下的正态均值矩阵，构造了推广James-Stein等经典估计量的极小极大估计量，提出推广Stein先验的超调和先验，经模拟验证其性能良好。

多元回归中高维残差协方差矩阵的受限非线性收缩

Restricted nonlinear shrinkage of high-dimensional residual covariance matrices in multivariate regressions

研究非正态模型的非线性收缩协方差估计，提出无边际非线性收缩估计器MENS，给出其经验谱分布收敛性及渐近最优性，建立相变，通过模拟和回测证实理论，展示其在高维配置和决策中的实用价值。

On the consistency of inversion-free parameter estimation for Gaussian random fields

Gaussian random fields are a powerful tool for modeling environmental processes. For high dimensional samples, classical approaches for estimating the covariance parameters require highly challenging and massive computations, such as the evaluation of the Cholesky factorization or solving linear systems. Recently, Anitescu, Chen and Stein \cite{M.Anitescu} proposed a fast and scalable algorithm which does not need such burdensome computations. The main focus of this article is to study the asymptotic behavior of the algorithm of Anitescu et al. (ACS) for regular and irregular grids in the incr…

Hilbert空间上线性高斯反问题的最优低秩近似，第一部分：后验协方差近似

Optimal low-rank approximations for linear Gaussian inverse problems on Hilbert spaces, Part I: posterior covariance approximation

本文利用Feldman-Hajek定理分析了无限维Hilbert空间上线性高斯反问题的后验协方差低秩近似，刻画了保证分布等价性的协方差近似条件，并在多种散度度量下同时求得最优低秩近似，为离散化问题提供了理论基础。

缓解集合数据同化中的方差损失：基于机器学习的无距离局部化方法

Mitigating loss of variance in ensemble data assimilation: machine learning-based and distance-free localization

本研究针对集合数据同化中采样误差导致的方差损失问题，提出两种适配表格数据的无距离局部化机器学习方法，集成至ES-MDA框架，可提升协方差估计与数据同化效果，且易实现、无需额外模拟或调参。

关于高维协方差阵最优线性收缩估计的Strong Convergence

On the Strong Convergence of the Optimal Linear Shrinkage Estimator for Large Dimensional Covariance Matrix

在这项工作中，我们构造了一个最佳的线性收缩估计的协方差阵在高维。随机矩阵理论的最新结果使我们能够找到最优收缩强度的渐近确定性等价物，并一致地估计它们。对于协方差阵，所得到的无分布估计在所有线性收缩估计中几乎必然服从最小Frobenius损失。我们考虑的情况包括变量数$p\rightarrow\infty$和样本量$n\rightarrow\infty$，因此$p/n\rightarrow c\in (0, +\infty)$。此外，我们证明了样本协方差阵的Frobenius范数几乎必然趋于一个确定的量，可以一致估计。

在高维中用于随机梯度估计的Stein规则收缩

Stein-Rule Shrinkage for Stochastic Gradient Estimation in High Dimensions

本文提出SR-Adam算法，通过Stein规则收缩改进高维随机梯度估计，在大规模学习中实现更优的性能。

使用数据驱动抖动的免调参一位协方差估计

Tuning-free one-bit covariance estimation using data-driven dithering

本文提出一种免调参的一位协方差估计方法，用数据驱动抖动替代需人工调节的参数 λ，并证明其非渐近误差界与调参版本及量化前样本协方差相当，仅多出对数损失。