The problem of obtaining optimal projections for performing discriminant analysis with Gaussian class densities is studied. Unlike in most existing approaches to the problem, the focus of the optimisation is on the multinomial likelihood based on posterior probability estimates, which directly captures discriminability of classes. In addition to the more commonly considered problem, in this context, of classification, the unsupervised clustering counterpart is also considered. Finding optimal projections offers utility for dimension reduction and regularisation, as well as instructive visualisation for better model interpretability. Practical applications of the proposed approach show considerable promise for both classification and clustering. Code to implement the proposed method is available in the form of an R package from https://github.com/DavidHofmeyr/OPGD.
In recent years, a considerable amount of work has been devoted to generalizing linear discriminant analysis to overcome its incompetence for high-dimensional classification (Witten & Tibshirani 2011, Cai & Liu 2011, Mai et al. 2012, Fan et al. 2012). In this paper, we develop high-dimensional semiparametric sparse discriminant analysis (HD-SeSDA) that generalizes the normal-theory discriminant analysis in two ways: it relaxes the Gaussian assumptions and can handle non-polynomial (NP) dimension classification problems. If the underlying Bayes rule is sparse, HD-SeSDA can estimate the Bayes ru…
Sequencing-based technologies provide an abundance of high-dimensional biological datasets with skewed and zero-inflated measurements. Classification of such data with linear discriminant analysis leads to poor performance due to the violation of the Gaussian distribution assumption. To address this limitation, we propose a new semiparametric discriminant analysis framework based on the truncated latent Gaussian copula model that accommodates both skewness and zero inflation. By applying sparsity regularization, we demonstrate that the proposed method leads to the consistent estimation of clas…
The focus of this paper is to extend Fisher's linear discriminant analysis (LDA) to both densely re-corded functional data and sparsely observed longitudinal data for general $c$-category classification problems. We propose an efficient approach to identify the optimal LDA projections in addition to managing the noninvertibility issue of the covariance operator emerging from this extension. A conditional expectation technique is employed to tackle the challenge of projecting sparse data to the LDA directions. We study the asymptotic properties of the proposed estimators and show that asymptoti…
The paper introduces a methodology for visualizing on a dimension reduced subspace the classification structure and the geometric characteristics induced by an estimated Gaussian mixture model for discriminant analysis. In particular, we consider the case of mixture of mixture models with varying parametrization which allow for parsimonious models. The approach is an extension of an existing work on reducing dimensionality for model-based clustering based on Gaussian mixtures. Information on the dimension reduction subspace is provided by the variation on class locations and, depending on the …
In this paper, we study high-dimensional sparse Quadratic Discriminant Analysis (QDA) and aim to establish the optimal convergence rates for the classification error. Minimax lower bounds are established to demonstrate the necessity of structural assumptions such as sparsity conditions on the discriminating direction and differential graph for the possible construction of consistent high-dimensional QDA rules. We then propose a classification algorithm called SDAR using constrained convex optimization under the sparsity assumptions. Both minimax upper and lower bounds are obtained and this cla…
This work presents a family of parsimonious Gaussian process models which allow to build, from a finite sample, a model-based classifier in an infinite dimensional space. The proposed parsimonious models are obtained by constraining the eigen-decomposition of the Gaussian processes modeling each class. This allows in particular to use non-linear mapping functions which project the observations into infinite dimensional spaces. It is also demonstrated that the building of the classifier can be directly done from the observation space through a kernel function. The proposed classification method…
University of Toronto（多伦多大学）；Cornell University（康奈尔大学）
This paper considers binary classification of high-dimensional features under a postulated model with a low-dimensional latent Gaussian mixture structure and non-vanishing noise. A generalized least squares estimator is used to estimate the direction of the optimal separating hyperplane. The estimated hyperplane is shown to interpolate on the training data. While the direction vector can be consistently estimated as could be expected from recent results in linear regression, a naive plug-in estimate fails to consistently estimate the intercept. A simple correction, that requires an independent…
Shape constrained regression analysis has applications in dose-response modeling, environmental risk assessment, disease screening and many other areas. Incorporating the shape constraints can improve estimation efficiency and avoid implausible results. We propose two novel methods focusing on Bayesian monotone curve and surface estimation using Gaussian process projections. The first projects samples from an unconstrained prior, while the second projects samples from the Gaussian process posterior. Theory is developed on continuity of the projection, posterior consistency and rates of contrac…
Discussion on "Random-projection ensemble classification" by T. Cannings and R. Samworth. We believe that the proposed approach can find many applications in economics such as credit scoring (e.g. Altman (1968)) and can be extended to more general type of classifiers. In this discussion we would like to draw authors attention to the copula-based discriminant analysis (Han et al. (2013) and He et al. (2016)).
A novel approach for supervised classification analysis for high dimensional and flat data (more variables than observations) is proposed. We use the information of class-membership of observations to determine groups of observations locally describing the group structure. By projecting the data on the subspace spanned by those groups, local projections are defined based on the projection concepts from Ortner et al. (2017a) and Ortner et al. (2017b). For each local projection a local discriminant analysis (LDA) model is computed using the information within the projection space as well as the …
Examples with bound information on the regression function and density abound in many real applications. We propose a novel approach for estimating such functions by incorporating the prior knowledge on the bounds. Specially, a Gaussian process is first imposed on the regression function whose posterior distribution is then projected onto the bounded space. The resulting projected measure is then used for inference. The projected sample path has closed form which facilitates efficient computations. In particular, our projection approach maintains a comparable computational efficiency with that…
We propose a new method for simplification of Gaussian process (GP) models by projecting the information contained in the full encompassing model and selecting a reduced number of variables based on their predictive relevance. Our results on synthetic and real world datasets show that the proposed method improves the assessment of variable relevance compared to the automatic relevance determination (ARD) via the length-scale parameters. We expect the method to be useful for improving explainability of the models, reducing the future measurement costs and reducing the computation time for makin…
Mixed Gaussian Projections for Two-Sample Testing of Functional Data
Centro de Matemática, Computação e Cognição, Universidade Federal do ABC (UFABC)（阿贝辛联邦大学数学、计算与认知中心）；Departamento de Matemática, Universidad de San Andrés–CONICET（圣安德烈斯大学数学系-阿根廷国家科学研究委员会）
A new characterization of random fields appearing in physical models is presented that is based on their well-known Homogeneous Chaos expansions. We take advantage of the adaptation capabilities of these expansions where the core idea is to rotate the basis of the underlying Gaussian Hilbert space, in order to achieve reduced functional representations that concentrate the induced probability measure in a lower dimensional subspace. For a smooth family of rotations along the domain of interest, the uncorrelated Gaussian inputs are transformed into a Gaussian process, thus introducing a mesosca…
We propose a projection pursuit (PP) algorithm based on Gaussian mixture models (GMMs). The negentropy obtained from a multivariate density estimated by GMMs is adopted as the PP index to be maximised. For a fixed dimension of the projection subspace, the GMM-based density estimation is projected onto that subspace, where an approximation of the negentropy for Gaussian mixtures is computed. Then, Genetic Algorithms (GAs) are used to find the optimal, orthogonal projection basis by maximising the former approximation. We show that this semi-parametric approach to PP is flexible and allows highl…
In this paper we address the statistical problem of testing if a stationary process is Gaussian. The observation consists in a finite sample path of the process. Using a random projection technique introduced and studied in Cuesta-Albertos et al. 2007 in the frame of goodness of fit test for functional data, we perform some decision rules. These rules really stand on the whole distribution of the process and not only on its marginal distribution at a fixed order. The main idea is to test the Gaussianity on the marginal distribution of some random linear combinations of the process. This leads …
It is now known that an extended Gaussian process model equipped with rescaling can adapt to different smoothness levels of a function valued parameter in many nonparametric Bayesian analyses, offering a posterior convergence rate that is optimal (up to logarithmic factors) for the smoothness class the true function belongs to. This optimal rate also depends on the dimension of the function's domain and one could potentially obtain a faster rate of convergence by casting the analysis in a lower dimensional subspace that does not amount to any loss of information about the true function. In gen…
