PCA is often used to visualize data when the rows and the columns are both of interest. In such a setting there is a lack of inferential methods on the PCA output. We study the asymptotic variance of a fixed-effects model for PCA, and propose several approaches to assessing the variability of PCA estimates: a method based on a parametric bootstrap, a new cell-wise jackknife, as well as a computationally cheaper approximation to the jackknife. We visualize the confidence regions by Procrustes rotation. Using a simulation study, we compare the proposed methods and highlight the strengths and drawbacks of each method as we vary the number of rows, the number of columns, and the strength of the relationships between variables.
Principal component analysis (PCA) is often used for analyzing data in the most diverse areas. In this work, we report an integrated approach to several theoretical and practical aspects of PCA. We start by providing, in an intuitive and accessible manner, the basic principles underlying PCA and its applications. Next, we present a systematic, though no exclusive, survey of some representative works illustrating the potential of PCA applications to a wide range of areas. An experimental investigation of the ability of PCA for variance explanation and dimensionality reduction is also developed,…
Inference for Heteroskedastic PCA with Missing Data
Institute for Data, Systems, and Society, MIT（麻省理工学院数据、系统与社会研究所）；Wharton School, University of Pennsylvania（宾夕法尼亚大学沃顿商学院）；Department of Operations Research and Financial Engineering, Princeton University（普林斯顿大学运筹学与金融工程系）
The Nyström method is one of the most popular techniques for improving the scalability of kernel methods. However, it has not yet been derived for kernel PCA in line with classical PCA. In this paper we derive kernel PCA with the Nyström method, thereby providing one of the few available options to make kernel PCA scalable. We further study its statistical accuracy through a finite-sample confidence bound on the empirical reconstruction error compared to the full method. The behaviours of the method and bound are illustrated through computer experiments on multiple real-world datasets. As an a…
In this note we consider PCA for Gaussian observations $X_1,\dots, X_n$ with covariance $Σ=\sum_i λ_i P_i$ in the 'effective rank' setting with model complexity governed by $\mathbf{r}(Σ):=\text{tr}(Σ)/\| Σ\|$. We prove a Berry-Essen type bound for a Wald Statistic of the spectral projector $\hat P_r$. This can be used to construct non-asymptotic confidence ellipsoids and tests for spectral projectors $P_r$. Using higher order pertubation theory we are able to show that our Theorem remains valid even when $\mathbf{r}(Σ) \gg \sqrt{n}$.
In the context of principal components analysis (PCA), the bootstrap is commonly applied to solve a variety of inference problems, such as constructing confidence intervals for the eigenvalues of the population covariance matrix $Σ$. However, when the data are high-dimensional, there are relatively few theoretical guarantees that quantify the performance of the bootstrap. Our aim in this paper is to analyze how well the bootstrap can approximate the joint distribution of the leading eigenvalues of the sample covariance matrix $\hatΣ$, and we establish non-asymptotic rates of approximation with…
In this paper, we study the power iteration algorithm for the spiked tensor model, as introduced in [44]. We give necessary and sufficient conditions for the convergence of the power iteration algorithm. When the power iteration algorithm converges, for the rank one spiked tensor model, we show the estimators for the spike strength and linear functionals of the signal are asymptotically Gaussian; for the multi-rank spiked tensor model, we show the estimators are asymptotically mixtures of Gaussian. This new phenomenon is different from the spiked matrix model. Using these asymptotic results of…
EPHE-PSL University（巴黎高等实践研究大学-巴黎文理研究大学）；Ho Chi Minh University of Technology（胡志明市技术大学）
Face recognition is the important field in machine learning and pattern recognition research area. It has a lot of applications in military, finance, public security, to name a few. In this paper, the combination of the tensor sparse PCA with the nearest-neighbor method (and with the kernel ridge regression method) will be proposed and applied to the face dataset. Experimental results show that the combination of the tensor sparse PCA with any classification system does not always reach the best accuracy performance measures. However, the accuracy of the combination of the sparse PCA method an…
Sparse principal component analysis (sPCA) has become one of the most widely used techniques for dimensionality reduction in high-dimensional datasets. The main challenge underlying sPCA is to estimate the first vector of loadings of the population covariance matrix, provided that only a certain number of loadings are non-zero. In this paper, we propose confidence intervals for individual loadings and for the largest eigenvalue of the population covariance matrix. Given an independent sample $X^i \in\mathbb R^p, i = 1,...,n,$ generated from an unknown distribution with an unknown covariance ma…
Factor analysis (FA) and principal component analysis (PCA) are popular statistical methods for summarizing and explaining the variability in multivariate datasets. By default, FA and PCA assume the number of components or factors to be known \emph{a priori}. However, in practice the users first estimate the number of factors or components and then perform FA and PCA analyses using the point estimate. Therefore, in practice the users ignore any uncertainty in the point estimate of the number of factors or components. For datasets where the uncertainty in the point estimate is not ignorable, it…
Network anomaly detection is a very relevant research area nowadays, especially due to its multiple applications in the field of network security. The boost of new models based on variational autoencoders and generative adversarial networks has motivated a reevaluation of traditional techniques for anomaly detection. It is, however, essential to be able to understand these new models from the perspective of the experience attained from years of evaluating network security data for anomaly detection. In this paper, we revisit anomaly detection techniques based on PCA from a probabilistic genera…
Sparse PCA (SPCA) is a fundamental model in machine learning and data analytics, which has witnessed a variety of application areas such as finance, manufacturing, biology, healthcare. To select a prespecified-size principal submatrix from a covariance matrix to maximize its largest eigenvalue for the better interpretability purpose, SPCA advances the conventional PCA with both feature selection and dimensionality reduction. This paper proposes two exact mixed-integer SDPs (MISDPs) by exploiting the spectral decomposition of the covariance matrix and the properties of the largest eigenvalues. …
Given a random sample from a multivariate population, estimating the number of large eigenvalues of the population covariance matrix is an important problem in Statistics with wide applications in many areas. In the context of Principal Component Analysis (PCA), the linear combinations of the original variables having the largest amounts of variation are determined by this number. In this paper, we study the high dimensional asymptotic regime where the number of variables grows at the same rate as the number of observations, and use the spiked covariance model proposed in Johnstone (2001), und…
Results. The PCA confirms the existence of the Gnevyshev gap (GG) for solar cycles at about 40% from the start of the cycle. The temporal evolution of sunspot area data for even cycles shows that the GG exists at least at the 95% confidence level for all sizes of sunspots. On the other hand, the GG is shorter and statistically insignificant for the odd cycles of aerial sunspot data. Furthermore, the analysis of sunspot area sizes for even and odd cycles of SC12-SC23 shows that the greatest difference is at 4.2-4.6 years, where even cycles have a far smaller total area than odd cycles. The aver…
In the analysis of survey data it is of interest to estimate and quantify uncertainty about means or totals for each of several non-overlapping subpopulations, or areas. When the sample size for a given area is small, standard confidence intervals based on data only from that area can be unacceptably wide. In order to reduce interval width, practitioners often utilize multilevel models in order to borrow information across areas, resulting in intervals centered around shrinkage estimators. However, such intervals only have the nominal coverage rate on average across areas under the assumed mod…
Empirical Bayes small area estimation based on the well-known Fay-Herriot model may produce unreliable estimates when outlying areas exist. Existing robust methods against outliers or model misspecification are generally inefficient when the assumed distribution is plausible. This paper proposes a simple modification of the standard empirical Bayes methods with adaptively balancing robustness and efficiency. The proposed method employs gamma-divergence instead of the marginal log-likelihood and optimizes a tuning parameter controlling robustness by pursuing the efficiency of empirical Bayes co…
Upper bounds on the minimum coverage probability of model averaged tail area confidence intervals in regression
频率主义模型平均已被提出作为将“模型不确定性”纳入置信区间构造的方法。环境和生态统计界对这些建议特别感兴趣。Turek和Fletcher在2012年提出的模型平均尾面积（MATA）置信区间是一种很有前途的方法。此区间的性能在很大程度上取决于它所基于的基于数据的模型权重。Kabaila，Welsh and Abeysekera，2016在两个嵌套线性回归模型的简单场景中提供了该区间覆盖概率的计算方便公式。我们考虑更复杂的情况，即有许多（在所考虑的示例中为32，768个）线性回归模型，如下所示。对于回归参数向量的一组指定分量中的每一个，我们要么将分量设置为零，要么让它自由变化。我们提供了一个易于计算的上限上的最小覆盖概率的MATA置信区间。该上限提供了反对使用基于贝叶斯信息准则（BIC）的模型权重的证据。
GEIRI North America（北美电力可靠性公司研究院）；State Grid Jiangsu Electric Power Company Ltd.（国网江苏省电力有限公司）；State Grid Nanjing Power Supply Company（国网南京供电公司）
Traditional load analysis is facing challenges with the new electricity usage patterns due to demand response as well as increasing deployment of distributed generations, including photovoltaics (PV), electric vehicles (EV), and energy storage systems (ESS). At the transmission system, despite of irregular load behaviors at different areas, highly aggregated load shapes still share similar characteristics. Load clustering is to discover such intrinsic patterns and provide useful information to other load applications, such as load forecasting and load modeling. This paper proposes an efficient…
Distributed Principal Component Analysis (PCA) has been studied to deal with the case when data are stored across multiple machines and communication cost or privacy concerns prohibit the computation of PCA in a central location. However, the sub-Gaussian assumption in the related literature is restrictive in real application where outliers or heavy-tailed data are common in areas such as finance and macroeconomic. In this article, we propose a distributed algorithm for estimating the principle eigenspaces without any moment constraint on the underlying distribution. We study the problem under…
The area under a receiver operating characteristic curve (AUC) is a useful tool to assess the performance of continuous-scale diagnostic tests on binary classification. In this article, we propose an empirical likelihood (EL) method to construct confidence intervals for the AUC from data collected by ranked set sampling (RSS). The proposed EL-based method enables inferences without assumptions required in existing nonparametric methods and takes advantage of the sampling efficiency of RSS. We show that for both balanced and unbalanced RSS, the EL-based point estimate is the Mann-Whitney statis…
A difficulty in MSE estimation occurs because we do not specify a full distribution for the survey weights. This obfuscates the use of fully parametric bootstrap procedures. To overcome this challenge, we develop a novel MSE estimator. We estimate the leading term in the MSE, which is the MSE of the best predictor (constructed with the true parameters), using the same simulated samples used to construct the basic predictor. We then exploit the asymptotic normal distribution of the parameter estimators to estimate the second term in the MSE, which reflects variability in the estimated parameter…
