Learning the mapping $\mathbf{x}\mapsto \sum_{i=1}^d x_i^2$: the cost of finding the needle in a haystack
University of California, Berkeley（加州大学伯克利分校）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Hong Kong University of Science and Technology（香港科技大学）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
Jiefu Zhang, Leonardo Zepeda-Núñez, Yuan Yao, Lin Lin
The task of using machine learning to approximate the mapping $\mathbf{x}\mapsto\sum_{i=1}^d x_i^2$ with $x_i\in[-1,1]$ seems to be a trivial one. Given the knowledge of the separable structure of the function, one can design a sparse network to represent the function very accurately, or even exactly. When such structural information is not available, and we may only use a dense neural network, the optimization procedure to find the sparse network embedded in the dense network is similar to finding the needle in a haystack, using a given number of samples of the function. We demonstrate that the cost (measured by sample complexity) of finding the needle is directly related to the Barron norm of the function. While only a small number of samples is needed to train a sparse network, the dense network trained with the same number of samples exhibits large test loss and a large generalization gap. In order to control the size of the generalization gap, we find that the use of explicit regularization becomes increasingly more important as $d$ increases. The numerically observed sample complexity with explicit regularization scales as $\mathcal{O}(d^{2.5})$, which is in fact better than the theoretically predicted sample complexity that scales as $\mathcal{O}(d^{4})$. Without explicit regularization (also called implicit regularization), the numerically observed sample complexity is significantly higher and is close to $\mathcal{O}(d^{4.5})$.
Shandong University（山东大学）；University of Geneva（日内瓦大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample size is smaller than the dimension of the input vector (haystack). More recently learners known as artificial neural networks (ANN) have shown great successes in many machine learning tasks, in particular fitting nonlinear associations. Small learning rate, stochastic gradient descent algorithm and large training set help to cope with the explosion in the number of parameters pre…
The increase in complexity of autonomous systems is accompanied by a need of data-driven development and validation strategies. Advances in computer graphics and cloud clusters have opened the way to massive parallel high fidelity simulations to qualitatively address the large number of operational scenarios. However, exploration of all possible scenarios is still prohibitively expensive and outcomes of scenarios are generally unknown apriori. To this end, the authors propose a method based on bayesian optimization to efficiently learn generative models on scenarios that would deliver desired …
Fast Bayesian Optimization of Needle-in-a-Haystack Problems using Zooming Memory-Based Initialization (ZoMBI)
Massachusetts Institute of Technology（麻省理工学院）；Singapore-MIT Alliance for Research and Technology（新加坡-麻省理工学院研究与技术联盟）；Xinterra；National University of Singapore（新加坡国立大学）
Many Needles in a Haystack: Active Hit Discovery for Perturbation Experiments
Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, UK（韦尔科姆桑格研究所，韦尔科姆基因组校园，英国辛顿）；Cambridge Center for AI in Medicine, University of Cambridge, Cambridge, UK（剑桥人工智能医学中心，剑桥大学，剑桥，英国）；Cambridge Stem Cell Institute, University of Cambridge, Cambridge, UK（剑桥干细胞研究所，剑桥大学，剑桥，英国）；Department of Computer Science and Technology, University of Cambridge, Cambridge, UK（剑桥计算机科学与技术系，剑桥大学，剑桥，英国）；MediaTek Research, Cambridge, UK（联发科研究，剑桥，英国）
Imperial College London（帝国理工学院）；University of Cambridge（剑桥大学）；King’s College London（伦敦国王学院）
Long-sequence transformers are designed to improve the representation of longer texts by language models and their performance on downstream document-level tasks. However, not much is understood about the quality of token-level predictions in long-form models. We investigate the performance of such architectures in the context of document classification with unsupervised rationale extraction. We find standard soft attention methods to perform significantly worse when combined with the Longformer language model. We propose a compositional soft attention architecture that applies RoBERTa sentenc…
University of York（约克大学）；University College London（伦敦大学学院）；Max Planck University College London Centre for Computational Psychiatry and Ageing Research（马克斯·普朗克伦敦大学学院计算精神病学与衰老研究中心）；University of Pennsylvania（宾夕法尼亚大学）；RWTH Aachen University（亚琛工业大学）；Jülich-Aachen Research Alliance（于利希-亚琛研究联盟）；INRIA（法国国家信息与自动化研究所）；CEA Saclay（法国原子能委员会萨克雷中心）
Since the beginning of the 21st century, the size, breadth, and granularity of data in biology and medicine has grown rapidly. In the example of neuroscience, studies with thousands of subjects are becoming more common, which provide extensive phenotyping on the behavioral, neural, and genomic level with hundreds of variables. The complexity of such big data repositories offer new opportunities and pose new challenges to investigate brain, cognition, and disease. Canonical correlation analysis (CCA) is a prototypical family of methods for wrestling with and harvesting insight from such rich da…
Important tasks like record linkage and extreme classification demonstrate extreme class imbalance, with 1 minority instance to every 1 million or more majority instances. Obtaining a sufficient sample of all classes, even just to achieve statistically-significant evaluation, is so challenging that most current approaches yield poor estimates or incur impractical cost. Where importance sampling has been levied against this challenge, restrictive constraints are placed on performance metrics, estimates do not come with appropriate guarantees, or evaluations cannot adapt to incoming labels. This…
University of Geneva（日内瓦大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
Using a sparsity inducing penalty in artificial neural networks (ANNs) avoids over-fitting, especially in situations where noise is high and the training set is small in comparison to the number of features. For linear models, such an approach provably also recovers the important features with high probability in regimes for a well-chosen penalty parameter. The typical way of setting the penalty parameter is by splitting the data set and performing the cross-validation, which is (1) computationally expensive and (2) not desirable when the data set is already small to be further split (for exam…
Recent results in coupled or temporal graphical models offer schemes for estimating the relationship structure between features when the data come from related (but distinct) longitudinal sources. A novel application of these ideas is for analyzing group-level differences, i.e., in identifying if trends of estimated objects (e.g., covariance or precision matrices) are different across disparate conditions (e.g., gender or disease). Often, poor effect sizes make detecting the differential signal over the full set of features difficult: for example, dependencies between only a subset of features…
Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find
Center for Alzheimer’s Disease and Related Dementias, NIA, NIH（阿尔茨海默病及相关痴呆症研究中心，国家老龄化研究所，国家卫生研究院）；DataTecnica LLC（DataTecnica公司）；Johns Hopkins University（约翰霍普金斯大学）；Laboratory of Neurogenetics, NIA, NIH（神经遗传学实验室，国家老龄化研究所，国家卫生研究院）
Transfer learning has been recently popularized as a data-efficient alternative to training models from scratch, in particular for computer vision tasks where it provides a remarkably solid baseline. The emergence of rich model repositories, such as TensorFlow Hub, enables the practitioners and researchers to unleash the potential of these models across a wide range of downstream tasks. As these repositories keep growing exponentially, efficiently selecting a good model for the task at hand becomes paramount. We provide a formalization of this problem through a familiar notion of regret and in…
Imperial College London（帝国理工学院）；Facebook AI Research（脸书人工智能研究院）；Radboud University Medical Center（拉德堡德大学医学中心）
In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional Neural Networks (CNNs) for image classification were developed using biased datasets that contain large objects, in mostly central image positions. To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one derived from MNIST digits and one from histopathology images. This testbed allows controlled experiments to stress-test CNN …
Instituto de Telecomunicações（电信研究所）；Instituto Superior Técnico（高等技术学院）；LUMLIS (Lisbon ELLIS Unit)（LUMLIS（里斯本ELLIS单元））；University of Edinburgh（爱丁堡大学）；Unbabel（Unbabel公司）
Although the problem of hallucinations in neural machine translation (NMT) has received some attention, research on this highly pathological phenomenon lacks solid ground. Previous work has been limited in several ways: it often resorts to artificial settings where the problem is amplified, it disregards some (common) types of hallucinations, and it does not validate adequacy of detection heuristics. In this paper, we set foundations for the study of NMT hallucinations. First, we work in a natural setting, i.e., in-domain data without artificial noise neither in training nor in inference. Next…
SCORE: A 1D Reparameterization Technique to Break Bayesian Optimization's Curse of Dimensionality
Laboratoire de Physique des Interfaces et des Couches Minces (LPICM)（界面与薄膜物理实验室（LPICM））；Ecole Polytechnique, Institut Polytechnique de Paris (IP Paris)（巴黎综合理工学院，巴黎理工学院（IP Paris））；Institut Photovoltaïque d’Île-de-France (IPVF)（法兰西岛光伏研究所（IPVF））
The research literature on cybersecurity incident detection & response is very rich in automatic detection methodologies, in particular those based on the anomaly detection paradigm. However, very little attention has been devoted to the diagnosis ability of the methods, aimed to provide useful information on the causes of a given detected anomaly. This information is of utmost importance for the security team to reduce the time from detection to response. In this paper, we present Multivariate Big Data Analysis (MBDA), a complete intrusion detection approach based on 5 steps to effectively ha…
Laboratoire de Physique de l’Ecole normale supérieure ENS（巴黎高等师范学院物理实验室）；Université PSL（巴黎文理研究大学）；CNRS（法国国家科学研究中心）；Sorbonne Université（索邦大学）；Université Paris-Diderot（巴黎第七大学）；Sorbonne Paris Cité（巴黎索邦西岱大学）；Facebook AI Research（脸书人工智能研究院）；Facebook（脸书）；Courant Institute of Mathematical Sciences and Center for Data Science（库朗数学科学研究所与数据科学中心）；New York University（纽约大学）
Despite the phenomenal success of deep neural networks in a broad range of learning tasks, there is a lack of theory to understand the way they work. In particular, Convolutional Neural Networks (CNNs) are known to perform much better than Fully-Connected Networks (FCNs) on spatially structured data: the architectural structure of CNNs benefits from prior knowledge on the features of the data, for instance their translation invariance. The aim of this work is to understand this fact through the lens of dynamics in the loss landscape. We introduce a method that maps a CNN to its equivalent FCN …
Materials discovery is often compared to the challenge of finding a needle in a haystack. While much work has focused on accurately predicting the properties of candidate materials with machine learning (ML), which amounts to evaluating whether a given candidate is a piece of straw or a needle, less attention has been paid to a critical question: Are we searching in the right haystack? We refer to the haystack as the design space for a particular materials discovery problem (i.e. the set of possible candidate materials to synthesize), and thus frame this question as one of design space selecti…
SLAC National Accelerator Laboratory（SLAC国家加速器实验室）；Utrecht University（乌得勒支大学）；Princeton University（普林斯顿大学）；CNRS（法国国家科学研究中心）；CEMES（材料结构与研究中心）；Tel Aviv University（特拉维夫大学）；Geophysical Institute of Israel（以色列地球物理研究所）
How does one search for a needle in a multi-dimensional haystack without knowing what a needle is and without knowing if there is one in the haystack? This kind of problem requires a paradigm shift - away from hypothesis driven searches of the data - towards a methodology that lets the data speak for itself. Dynamic Quantum Clustering (DQC) is such a methodology. DQC is a powerful visual method that works with big, high-dimensional data. It exploits variations of the density of the data (in feature space) and unearths subsets of the data that exhibit correlations among all the measured variabl…
Detecting Scarce and Sparse Anomalous: Solving Dual Imbalance in Multi-Instance Learning
National Key Laboratory for Novel Software Technology（国家新型软件技术重点实验室）；School of Artificial Intelligence（人工智能学院）；School of Intelligence Science and Technology（智能科学与技术学院）；Nanjing University（南京大学）；Didi Chuxing（滴滴出行）
