Local intrinsic dimensionality estimators based on concentration of measure
Institut Curie（居里研究所）；PSL Research University（巴黎文理研究大学）；Mines ParisTech（巴黎高等矿业学院）；INSERM（法国国家健康与医学研究院）；Lobachevsky University（罗巴切夫斯基大学）
Jonathan Bac, Andrei Zinovyev
备注：
to be published in the International Joint Conference On Neural Networks (IJCNN) held as part of the IEEE World Congress On Computational Intelligence (WCCI), July 2020
Intrinsic dimensionality (ID) is one of the most fundamental characteristics of multi-dimensional data point clouds. Knowing ID is crucial to choose the appropriate machine learning approach as well as to understand its behavior and validate it. ID can be computed globally for the whole data point distribution, or computed locally in different regions of the data space. In this paper, we introduce new local estimators of ID based on linear separability of multi-dimensional data point clouds, which is one of the manifestations of concentration of measure. We empirically study the properties of these estimators and compare them with other recently introduced ID estimators exploiting various effects of measure concentration. Observed differences between estimators can be used to anticipate their behaviour in practical applications.
In the last decades the estimation of the intrinsic dimensionality of a dataset has gained considerable importance. Despite the great deal of research work devoted to this task, most of the proposed solutions prove to be unreliable when the intrinsic dimensionality of the input dataset is high and the manifold where the points lie is nonlinearly embedded in a higher dimensional space. In this paper we propose a novel robust intrinsic dimensionality estimator that exploits the twofold complementary information conveyed both by the normalized nearest neighbor distances and by the angles computed…
Mutual information and task-relevant latent dimensionality
Department of Physics, Initiative for Theory and Modeling of Living Systems, Emory University（物理系、生命系统理论与建模倡议、埃默里大学）；School of Physics, School of Psychology, Georgia Institute of Technology（物理学院、心理学学院、佐治亚理工学院）；Department of Biology, Emory University（生物学系、埃默里大学）
The intrinsic dimensionality refers to the ``true'' dimensionality of the data, as opposed to the dimensionality of the data representation. For example, when attributes are highly correlated, the intrinsic dimensionality can be much lower than the number of variables. Local intrinsic dimensionality refers to the observation that this property can vary for different parts of the data set; and intrinsic dimensionality can serve as a proxy for the local difficulty of the data set. Most popular methods for estimating the local intrinsic dimensionality are based on distances, and the rate at which…
Wigner Research Centre for Physics（维格纳物理研究中心）；Semmelweis University（塞梅尔维斯大学）；National Institute of Clinical Neurosciences（国家临床神经科学研究所）；Péter Pázmány Catholic University（帕兹马尼·彼得天主教大学）；Budapest University of Technology and Economics（布达佩斯技术与经济大学）；University of Pannonia（潘诺尼亚大学）
Data dimensionality informs us about data complexity and sets limit on the structure of successful signal processing pipelines. In this work we revisit and improve the manifold-adaptive Farahmand-Szepesvári-Audibert (FSA) dimension estimator, making it one of the best nearest neighbor-based dimension estimators available. We compute the probability density function of local FSA estimates, if the local manifold density is uniform. Based on the probability density function, we propose to use the median of local estimates as a basic global measure of intrinsic dimensionality, and we demonstrate t…
University of Leicester（莱斯特大学）；Saint-Petersburg State Electrotechnical University（圣彼得堡国立电子技术大学）
The concentration of measure phenomena were discovered as the mathematical background of statistical mechanics at the end of the XIX - beginning of the XX century and were then explored in mathematics of the XX-XXI centuries. At the beginning of the XXI century, it became clear that the proper utilisation of these phenomena in machine learning might transform the curse of dimensionality into the blessing of dimensionality. This paper summarises recently discovered phenomena of measure concentration which drastically simplify some machine learning problems in high dimension, and allow us to cor…
New Jersey Institute of Technology（新泽西理工学院）；Bremen University of Applied Sciences（不来梅应用技术大学）；Princeton University（普林斯顿大学）；Technion - Israel Institute of Technology（以色列理工学院）
A common way of characterizing minimax estimators in point estimation is by moving the problem into the Bayesian estimation domain and finding a least favorable prior distribution. The Bayesian estimator induced by a least favorable prior, under mild conditions, is then known to be minimax. However, finding least favorable distributions can be challenging due to inherent optimization over the space of probability distributions, which is infinite-dimensional. This paper develops a dimensionality reduction method that allows us to move the optimization to a finite-dimensional setting with an exp…
The Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）
The failure of the Euclidean norm to reliably distinguish between nearby and distant points in high dimensional space is well-known. This phenomenon of distance concentration manifests in a variety of data distributions, with iid or correlated features, including centrally-distributed and clustered data. Unsupervised learning based on Euclidean nearest-neighbors and more general proximity-oriented data mining tasks like clustering, might therefore be adversely affected by distance concentration for high-dimensional applications. While considerable work has been done developing clustering algor…
The University of Tokyo（东京大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；Kyushu Institute of Technology（九州工业大学）
Deep learning has exhibited superior performance for various tasks, especially for high-dimensional datasets, such as images. To understand this property, we investigate the approximation and estimation ability of deep learning on anisotropic Besov spaces. The anisotropic Besov space is characterized by direction-dependent smoothness and includes several function classes that have been investigated thus far. We demonstrate that the approximation error and estimation error of deep learning only depend on the average value of the smoothness parameters in all directions. Consequently, the curse o…
CNRS-IRISA（法国国家科学研究中心-法国信息与自动化研究所）；Amazon Web Services（亚马逊云科技）；The University of Melbourne（墨尔本大学）；National Institute of Informatics（国立信息学研究所）；University of Novi Sad, Faculty of Sciences（诺维萨德大学理学院）；Bank of Thailand（泰国银行）
Accurate estimation of Intrinsic Dimensionality (ID) is of crucial importance in many data mining and machine learning tasks, including dimensionality reduction, outlier detection, similarity search and subspace clustering. However, since their convergence generally requires sample sizes (that is, neighborhood sizes) on the order of hundreds of points, existing ID estimation methods may have only limited usefulness for applications in which the data consists of many natural groups of small size. In this paper, we propose a local ID estimation strategy stable even for `tight' localities consist…
This paper deals with a new filter algorithm for selecting the smallest subset of features carrying all the information content of a data set (i.e. for removing redundant features). It is an advanced version of the fractal dimension reduction technique, and it relies on the recently introduced Morisita estimator of Intrinsic Dimension (ID). Here, the ID is used to quantify dependencies between subsets of features, which allows the effective processing of highly non-linear data. The proposed algorithm is successfully tested on simulated and real world case studies. Different levels of sample si…
Rutgers University（罗格斯大学）；The University of Tokyo（东京大学）；Komaba Institute for Science, The University of Tokyo（东京大学驹场科学研究所）；Center for Advanced Intelligence Project, RIKEN（理化学研究所革新智能综合研究中心）
In this study, we prove that an intrinsic low dimensionality of covariates is the main factor that determines the performance of deep neural networks (DNNs). DNNs generally provide outstanding empirical performance. Hence, numerous studies have actively investigated the theoretical properties of DNNs to understand their underlying mechanisms. In particular, the behavior of DNNs in terms of high-dimensional data is one of the most critical questions. However, this issue has not been sufficiently investigated from the aspect of covariates, although high-dimensional data have practically low intr…
University of California, Berkeley（加利福尼亚大学伯克利分校）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University College London（伦敦大学学院）
Density estimation is a fundamental problem in statistical learning. This problem is especially challenging for complex high-dimensional data due to the curse of dimensionality. A promising solution to this problem is given here in an inference-free hierarchical framework that is built on score matching. We revisit the Bayesian interpretation of the score function and the Parzen score matching, and construct a multilayer perceptron with a scalable objective for learning the energy (i.e. the unnormalized log-density), which is then optimized with stochastic gradient descent. In addition, the re…
University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
We study the problem of estimating an unknown function from noisy data using shallow ReLU neural networks. The estimators we study minimize the sum of squared data-fitting errors plus a regularization term proportional to the squared Euclidean norm of the network weights. This minimization corresponds to the common approach of training a neural network with weight decay. We quantify the performance (mean-squared error) of these neural network estimators when the data-generating function belongs to the second-order Radon-domain bounded variation space. This space of functions was recently propo…
Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；Center for Computational Mathematics, Flatiron Institute（熨斗研究所计算数学中心）；Inria（法国国家信息与自动化研究所）；Ecole Normale Supérieure（巴黎高等师范学院）；PSL Research University（巴黎文理研究大学）
Spectral clustering and diffusion maps are celebrated dimensionality reduction algorithms built on eigen-elements related to the diffusive structure of the data. The core of these procedures is the approximation of a Laplacian through a graph kernel approach, however this local average construction is known to be cursed by the high-dimension d. In this article, we build a different estimator of the Laplacian, via a reproducing kernel Hilbert space method, which adapts naturally to the regularity of the problem. We provide non-asymptotic statistical rates proving that the kernel estimator we bu…
Dalhousie University（达尔豪斯大学）；University of Science（理大学）；Vietnam National University（越南国立大学）；AISIA Research Lab（AISIA研究实验室）；University of Delaware（特拉华大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
In this paper, we study the learning rate of generalized Bayes estimators in a general setting where the hypothesis class can be uncountable and have an irregular shape, the loss function can have heavy tails, and the optimal hypothesis may not be unique. We prove that under the multi-scale Bernstein's condition, the generalized posterior distribution concentrates around the set of optimal hypotheses and the generalized Bayes estimator can achieve fast learning rate. Our results are applied to show that the standard Bayesian linear regression is robust to heavy-tailed distributions.
Advanced Research Institute of Multidisciplinary Science (ARIMS)（多学科科学高级研究院（ARIMS））；School of Information and Electronics, Beijing Institute of Technology (BIT)（北京理工大学信息与电子学院）
In this paper, we propose a data-driven deep learning (DL) approach to jointly design the pilot signals and channel estimator for wideband massive multiple-input multiple-output (MIMO) systems. By exploiting the angular-domain compressibility of massive MIMO channels, the conceived DL framework can reliably reconstruct the high-dimensional channels from the under-determined measurements. Specifically, we design an end-to-end deep neural network (DNN) architecture composed of dimensionality reduction network and reconstruction network to respectively mimic the pilot signals and channel estimato…
Dimensionality-Aware Outlier Detection: Theoretical and Experimental Analysis
The University of Newcastle（纽卡斯尔大学）；The University of Melbourne（墨尔本大学）；University of Southern Denmark（南丹麦大学）；New Jersey Institute of Technology（新泽西理工学院）；University of Novi Sad（诺威萨大学）
A general framework for adaptive nonparametric dimensionality reduction
Seminar for Statistics, Department of Mathematics, ETH Zurich（苏黎世联邦理工学院统计研究所，数学系）；Faculty of Economics, Euler Institute, Università della Svizzera italiana（瑞士意大利大学经济学院，欧拉研究所）；Faculty of Informatics, Università della Svizzera italiana（瑞士意大利大学信息学院）；Department of Science and High Technology, University of Insubria（意大利因布里亚大学科学与高科技系）
Technische Universität Berlin（柏林工业大学）；Machine Learning Group（机器学习组）
While nonparametric density estimators often perform well on low dimensional data, their performance can suffer when applied to higher dimensional data, owing presumably to the curse of dimensionality. One technique for avoiding this is to assume no dependence between features and that the data are sampled from a separable density. This allows one to estimate each marginal distribution independently thereby avoiding the slow rates associated with estimating the full joint density. This is a strategy employed in naive Bayes models and is analogous to estimating a rank-one tensor. In this paper …
The von Mises-Fisher (vMF) is a well-known density model for directional random variables. The recent surge of the deep embedding methodologies for high-dimensional structured data such as images or texts, aimed at extracting salient directional information, can make the vMF model even more popular. In this article, we will review the vMF model and its mixture, provide detailed recipes of how to train the models, focusing on the maximum likelihood estimators, in Python/PyTorch. In particular, implementation of vMF typically suffers from the notorious numerical issue of the Bessel function eval…
A UCB Bandit Algorithm for General ML-Based Estimators
School of Mathematics, University of Minnesota—Twin Cities（明尼苏达大学 Twin Cities 分校数学系）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
