The concentration of measure phenomena were discovered as the mathematical background of statistical mechanics at the end of the XIX - beginning of the XX century and were then explored in mathematics of the XX-XXI centuries. At the beginning of the XXI century, it became clear that the proper utilisation of these phenomena in machine learning might transform the curse of dimensionality into the blessing of dimensionality.
This paper summarises recently discovered phenomena of measure concentration which drastically simplify some machine learning problems in high dimension, and allow us to correct legacy artificial intelligence systems. The classical concentration of measure theorems state that i.i.d. random points are concentrated in a thin layer near a surface (a sphere or equators of a sphere, an average or median level set of energy or another Lipschitz function, etc.).
The new stochastic separation theorems describe the thin structure of these thin layers: the random points are not only concentrated in a thin layer but are all linearly separable from the rest of the set, even for exponentially large random sets. The linear functionals for separation of points can be selected in the form of the linear Fisher's discriminant.
All artificial intelligence systems make errors. Non-destructive correction requires separation of the situations (samples) with errors from the samples corresponding to correct behaviour by a simple and robust classifier. The stochastic separation theorems provide us by such classifiers and a non-iterative (one-shot) procedure for learning.
University of Leicester（莱斯特大学）；Lobachevsky University（罗巴切夫斯基大学）；Universidad Complutense de Madrid（马德里康普顿斯大学）
High-dimensional data and high-dimensional representations of reality are inherent features of modern Artificial Intelligence systems and applications of machine learning. The well-known phenomenon of the "curse of dimensionality" states: many problems become exponentially difficult in high dimensions. Recently, the other side of the coin, the "blessing of dimensionality", has attracted much attention. It turns out that generic high-dimensional datasets exhibit fairly simple geometric properties. Thus, there is a fundamental tradeoff between complexity and simplicity in high dimensional spaces…
University of Leicester（莱斯特大学）；Lobachevsky University（罗巴切夫斯基大学）；St Petersburg State Electrotechnical University（圣彼得堡国立电工大学）；University College London（伦敦大学学院）；Northeastern University（东北大学）
In this paper we present theory and algorithms enabling classes of Artificial Intelligence (AI) systems to continuously and incrementally improve with a-priori quantifiable guarantees - or more specifically remove classification errors - over time. This is distinct from state-of-the-art machine learning, AI, and software approaches. Another feature of this approach is that, in the supervised setting, the computational complexity of training is linear in the number of training samples. At the time of classification, the computational complexity is bounded by few inner product calculations. More…
Blind denoising diffusion models and the blessings of dimensionality
Flatiron Institute, Simons Foundation（Flatiron研究院，Simons基金会）；Foundations of Data Science, Yale University（数据科学基础，耶鲁大学）；Department of Statistics and Data Science, Yale University（统计与数据科学系，耶鲁大学）；Ctr. for Neural Science & Courant Institute, New York University（神经科学中心及Courant学院，纽约大学）
From field-scale to large-scale spectral libraries: Tabular foundation models in soil spectroscopy
Osnabrück University（奥斯纳布吕克大学）；Leibniz Institute for Agricultural Engineering and Bioeconomy (ATB)（莱布尼茨农业工程与生物经济研究所）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
Probabilistic Foundations of Fuzzy Simplicial Sets for Nonlinear Dimensionality Reduction
Max Planck Institute for Mathematics in the Sciences（马克斯·普朗克数学科学研究所）；Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；Max Planck School of Cognition（马克斯·普朗克认知学校）；ScaDS.AI (Center for Scalable Data Analytics and AI)（ScaDS.AI（可扩展数据与人工智能中心））；Santa Fe Institute for the Sciences of Complexity（复杂性科学圣菲研究所）
Manifold learning is a popular and quickly-growing subfield of machine learning based on the assumption that one's observed data lie on a low-dimensional manifold embedded in a higher-dimensional space. This thesis presents a mathematical perspective on manifold learning, delving into the intersection of kernel learning, spectral graph theory, and differential geometry. Emphasis is placed on the remarkable interplay between graphs and manifolds, which forms the foundation for the widely-used technique of manifold regularization. This work is written to be accessible to a broad mathematical aud…
Tadpole: Autoencoders as Foundation Models for 3D PDEs with Online Learning
TUM School of Computation, Information and Technology（慕尼黑技术大学计算、信息与技术学院）；Technical University of Munich, Garching, Germany（慕尼黑技术大学，慕尼黑，德国）；MCML, Munich Center for Machine Learning, Munich, Germany（慕尼黑机器学习中心，慕尼黑，德国）
Dimension reduction of data sets is a standard problem in the realm of machine learning and knowledge reasoning. They affect patterns in and dependencies on data dimensions and ultimately influence any decision-making processes. Therefore, a wide variety of reduction procedures are in use, each pursuing different objectives. A so far not considered criterion is the conceptual continuity of the reduction mapping, i.e., the preservation of the conceptual structure with respect to the original data set. Based on the notion scale-measure from formal concept analysis we present in this work a) the …
DaSCI, Andalusian Research Institute in Data Science and Computational Intelligence（DaSCI 安达卢西亚数据科学与计算智能研究所）；University of Granada（格拉纳达大学）
Distance metric learning is a branch of machine learning that aims to learn distances from the data, which enhances the performance of similarity-based algorithms. This tutorial provides a theoretical background and foundations on this topic and a comprehensive experimental analysis of the most-known algorithms. We start by describing the distance metric learning problem and its main mathematical foundations, divided into three main blocks: convex analysis, matrix analysis and information theory. Then, we will describe a representative set of the most popular distance metric learning methods u…
University of Virginia（弗吉尼亚大学）；University of Minnesota（明尼苏达大学）；Amazon（亚马逊）
Effective non-parametric density estimation is a key challenge in high-dimensional multivariate data analysis. In this paper,we propose a novel approach that builds upon tensor factorization tools. Any multivariate density can be represented by its characteristic function, via the Fourier transform. If the sought density is compactly supported, then its characteristic function can be approximated, within controllable error, by a finite tensor of leading Fourier coefficients, whose size de-pends on the smoothness of the underlying density. This tensor can be naturally estimated from observed re…
University of Washington（华盛顿大学）；Georgia Institute of Technology（佐治亚理工学院）
Dimensionality reduction is a classical technique widely used for data analysis. One foundational instantiation is Principal Component Analysis (PCA), which minimizes the average reconstruction error. In this paper, we introduce the "multi-criteria dimensionality reduction" problem where we are given multiple objectives that need to be optimized simultaneously. As an application, our model captures several fairness criteria for dimensionality reduction such as our novel Fair-PCA problem and the Nash Social Welfare (NSW) problem. In Fair-PCA, the input data is divided into $k$ groups, and the g…
Beyond IID: How General Are Tabular Foundation Models, Really?
Prior Labs；University of Freiburg（弗赖堡大学）；University of Mannheim（曼海姆大学）；INRIA Saclay（法国国家信息与自动化研究所萨克雷研究中心）；Technion（技术学院）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Zuse School ELIZA（Zuse学校ELIZA）
This survey is written in summer, 2016. The purpose of this survey is to briefly introduce nonlinear dimensionality reduction (NLDR) in data reduction. The first two NLDR were respectively published in Science in 2000 in which they solve the similar reduction problem of high-dimensional data endowed with the intrinsic nonlinear structure. The intrinsic nonlinear structure is always interpreted as a concept in manifolds from geometry and topology in theoretical mathematics by computer scientists and theoretical physicists. In 2001, the concept of Manifold Learning first appears as an NLDR metho…
University of Konstanz（康斯坦茨大学）；University of Stuttgart（斯图加特大学）
We present a technique to perform dimensionality reduction on data that is subject to uncertainty. Our method is a generalization of traditional principal component analysis (PCA) to multivariate probability distributions. In comparison to non-linear methods, linear dimensionality reduction techniques have the advantage that the characteristics of such probability distributions remain intact after projection. We derive a representation of the PCA sample covariance matrix that respects potential uncertainty in each of the inputs, building the mathematical foundation of our new method: uncertain…
ChromFound: Towards A Universal Foundation Model for Single-Cell Chromatin Accessibility Data
Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化院）；Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）；Computer, Electrical and Mathematical Sciences and Engineering Division, KAUST（卡塔尔大学计算机、电子与数学科学与工程系）；Center of Excellence for Smart Health, KAUST（卡塔尔大学智能健康卓越中心）；Center of Excellence on GenAI, KAUST（卡塔尔大学生成人工智能卓越中心）；Zhongshan Hospital, Fudan University（复旦大学中山医院）
A Multimodal 3D Foundation Model for Light Sheet Fluorescence Microscopy Enables Few-Shot Segmentation, Classification, and Deblurring
Tri-Institutional Program in Computational Biology \& Medicine, Weill Cornell Medicine, New York, NY, USA Department of Radiology, Weill Cornell Medicine, New York, NY, USA Helen；Robert Appel Alzheimers Disease Research Institute, Feil Family Brain；Mind Research Institute, Weill Cornell Medicine, New York, NY, USA Graduate Program in Physiology, Biophysics；Systems Biology, Weill Cornell Medicine, New York, NY, USA Cornell Tech, New York, NY, USA Institute for Intelligent Biotechnologies (iBIO), Helmholtz Center Munich, Neuherberg, Germany Institute for Stroke；Dementia Research, Klinikum der Universität München, Ludwig-Maximilians University Munich, Munich, Germany
