arXiv 2201.086522022-01-21stat.MLcs.LG

A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks

Shandong University（山东大学）；University of Geneva（日内瓦大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）

Xiaoyu Ma, Sylvain Sardy, Nick Hengartner, Nikolai Bobenko, Yen Ting Lin

英文摘要：

To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample size is smaller than the dimension of the input vector (haystack). More recently learners known as artificial neural networks (ANN) have shown great successes in many machine learning tasks, in particular fitting nonlinear associations. Small learning rate, stochastic gradient descent algorithm and large training set help to cope with the explosion in the number of parameters present in deep neural networks. Yet few ANN learners have been developed and studied to find needles in nonlinear haystacks. Driven by a single hyperparameter, our ANN learner, like for sparse linear associations, exhibits a phase transition in the probability of retrieving the needles, which we do not observe with other ANN learners. To select our penalty parameter, we generalize the universal threshold of Donoho and Johnstone (1994) which is a better rule than the conservative (too many false detections) and expensive cross-validation. In the spirit of simulated annealing, we propose a warm-start sparsity inducing algorithm to solve the high-dimensional, non-convex and non-differentiable optimization problem. We perform precise Monte Carlo simulations to show the effectiveness of our approach.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Finding Needles in Haystack: Formal Generative Models for Efficient Massive Parallel Simulations

RWTH Aachen University（亚琛工业大学）

The increase in complexity of autonomous systems is accompanied by a need of data-driven development and validation strategies. Advances in computer graphics and cloud clusters have opened the way to massive parallel high fidelity simulations to qualitatively address the large number of operational scenarios. However, exploration of all possible scenarios is still prohibitively expensive and outcomes of scenarios are generally unknown apriori. To this end, the authors propose a method based on bayesian optimization to efficiently learn generative models on scenarios that would deliver desired …

2023-01-03cs.LGcs.SYeess.SY

Learning the mapping $\mathbf{x}\mapsto \sum_{i=1}^d x_i^2$: the cost of finding the needle in a haystack

University of California, Berkeley（加州大学伯克利分校）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Hong Kong University of Science and Technology（香港科技大学）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）

The task of using machine learning to approximate the mapping $\mathbf{x}\mapsto\sum_{i=1}^d x_i^2$ with $x_i\in[-1,1]$ seems to be a trivial one. Given the knowledge of the separable structure of the function, one can design a sparse network to represent the function very accurately, or even exactly. When such structural information is not available, and we may only use a dense neural network, the optimization procedure to find the sparse network embedded in the dense network is similar to finding the needle in a haystack, using a given number of samples of the function. We demonstrate that t…

2020-05-16cs.LGcs.NAmath.NA

使用缩放记忆初始化（ZoMBI）快速进行贝叶斯优化以解决大海捞针问题

Fast Bayesian Optimization of Needle-in-a-Haystack Problems using Zooming Memory-Based Initialization (ZoMBI)

Massachusetts Institute of Technology（麻省理工学院）；Singapore-MIT Alliance for Research and Technology（新加坡-麻省理工学院研究与技术联盟）；Xinterra；National University of Singapore（新加坡国立大学）

提出缩放记忆初始化算法ZoMBI，通过迭代缩小搜索边界并修剪历史记忆，结合自适应采集函数，解决了多维大海捞针优化问题，计算速度提升400倍且优化程度高出基线3倍。

2023-02-03cond-mat.mtrl-scics.LGmath.OC

Model selection with lasso-zero: adding straw to the haystack to better find needles

The high-dimensional linear model $y = X β^0 + ε$ is considered and the focus is put on the problem of recovering the support $S^0$ of the sparse vector $β^0.$ We introduce Lasso-Zero, a new $\ell_1$-based estimator whose novelty resides in an "overfit, then threshold" paradigm and the use of noise dictionaries concatenated to $X$ for overfitting the response. To select the threshold, we employ the quantile universal threshold based on a pivotal statistic that requires neither knowledge nor preliminary estimation of the noise level. Numerical simulations show that Lasso-Zero performs well in t…

2019-04-12stat.MEstat.ML

haystack 中的许多针：用于扰动实验的主动命中发现

Many Needles in a Haystack: Active Hit Discovery for Perturbation Experiments

Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, UK（韦尔科姆桑格研究所，韦尔科姆基因组校园，英国辛顿）；Cambridge Center for AI in Medicine, University of Cambridge, Cambridge, UK（剑桥人工智能医学中心，剑桥大学，剑桥，英国）；Cambridge Stem Cell Institute, University of Cambridge, Cambridge, UK（剑桥干细胞研究所，剑桥大学，剑桥，英国）；Department of Computer Science and Technology, University of Cambridge, Cambridge, UK（剑桥计算机科学与技术系，剑桥大学，剑桥，英国）；MediaTek Research, Cambridge, UK（联发科研究，剑桥，英国）

本文提出Probability-of-Hit方法，通过后验概率排名候选者，有效发现超过阈值的扰动，提升实验效率。

2026-05-11cs.LG

大海捞针：面向贝叶斯优化的测试时模拟电路表示自适应

Finding the Needle in a Haystack: Test-Time Analog Circuit Representation Adaptation for Bayesian Optimization

North Carolina State University（北卡罗来纳州立大学）

本文提出TTARO框架，通过测试时自适应调整电路表示优化贝叶斯优化，在40种设置下较传统方法降低后悔曲线下面积15.2%至20.7%，性能优于现有技术。

2026-08-14cs.LG

更多相关论文收起更多论文

Finding Differentially Covarying Needles in a Temporally Evolving Haystack: A Scan Statistics Perspective

University of Wisconsin-Madison（威斯康星大学麦迪逊分校）

Recent results in coupled or temporal graphical models offer schemes for estimating the relationship structure between features when the data come from related (but distinct) longitudinal sources. A novel application of these ideas is for analyzing group-level differences, i.e., in identifying if trends of estimated objects (e.g., covariance or precision matrices) are different across disparate conditions (e.g., gender or disease). Often, poor effect sizes make detecting the differential signal over the full set of features difficult: for example, dependencies between only a subset of features…

2017-11-20cs.LGstat.ML

Finding the needle in high-dimensional haystack: A tutorial on canonical correlation analysis

University of York（约克大学）；University College London（伦敦大学学院）；Max Planck University College London Centre for Computational Psychiatry and Ageing Research（马克斯·普朗克伦敦大学学院计算精神病学与衰老研究中心）；University of Pennsylvania（宾夕法尼亚大学）；RWTH Aachen University（亚琛工业大学）；Jülich-Aachen Research Alliance（于利希-亚琛研究联盟）；INRIA（法国国家信息与自动化研究所）；CEA Saclay（法国原子能委员会萨克雷中心）

Since the beginning of the 21st century, the size, breadth, and granularity of data in biology and medicine has grown rapidly. In the example of neuroscience, studies with thousands of subjects are becoming more common, which provide extensive phenotyping on the behavioral, neural, and genomic level with hundreds of variables. The complexity of such big data repositories offer new opportunities and pose new challenges to investigate brain, cognition, and disease. Canonical correlation analysis (CCA) is a prototypical family of methods for wrestling with and harvesting insight from such rich da…

2018-12-06cs.LGstat.APstat.ML

Needle in a Haystack: Label-Efficient Evaluation under Extreme Class Imbalance

University of Melbourne（墨尔本大学）

Important tasks like record linkage and extreme classification demonstrate extreme class imbalance, with 1 minority instance to every 1 million or more majority instances. Obtaining a sufficient sample of all classes, even just to achieve statistically-significant evaluation, is so challenging that most current approaches yield poor estimates or incur impractical cost. Where importance sampling has been levied against this challenge, restrictive constraints are placed on performance metrics, estimates do not come with appropriate guarantees, or evaluations cannot adapt to incoming labels. This…

2021-06-02cs.IRcs.LGstat.ML

What needles do sparse neural networks find in nonlinear haystacks

University of Geneva（日内瓦大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）

Using a sparsity inducing penalty in artificial neural networks (ANNs) avoids over-fitting, especially in situations where noise is high and the training set is small in comparison to the number of features. For linear models, such an approach provably also recovers the important features with high probability in regimes for a well-chosen penalty parameter. The typical way of setting the penalty parameter is by splitting the data set and performing the cross-validation, which is (1) computationally expensive and (2) not desirable when the data set is already small to be further split (for exam…

2020-06-07cs.LGphysics.data-anstat.ML

Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias

Laboratoire de Physique de l’Ecole normale supérieure ENS（巴黎高等师范学院物理实验室）；Université PSL（巴黎文理研究大学）；CNRS（法国国家科学研究中心）；Sorbonne Université（索邦大学）；Université Paris-Diderot（巴黎第七大学）；Sorbonne Paris Cité（巴黎索邦西岱大学）；Facebook AI Research（脸书人工智能研究院）；Facebook（脸书）；Courant Institute of Mathematical Sciences and Center for Data Science（库朗数学科学研究所与数据科学中心）；New York University（纽约大学）

Despite the phenomenal success of deep neural networks in a broad range of learning tasks, there is a lack of theory to understand the way they work. In particular, Convolutional Neural Networks (CNNs) are known to perform much better than Fully-Connected Networks (FCNs) on spatially structured data: the architectural structure of CNNs benefits from prior knowledge on the features of the data, for instance their translation invariance. The aim of this work is to understand this fact through the lens of dynamics in the loss landscape. We introduce a method that maps a CNN to its equivalent FCN …

2020-02-04cond-mat.dis-nncond-mat.stat-mechcs.LG