Linear Recursive Feature Machines provably recover low-rank matrices
Harvard University（哈佛大学）；Broad Institute of MIT and Harvard（布罗德研究所）；Halıcıoğlu Data Science Institute, UC San Diego（加州大学圣迭戈分校哈勒西奥卢数据科学研究所）；University of Washington（华盛顿大学）
A fundamental problem in machine learning is to understand how neural networks make accurate predictions, while seemingly bypassing the curse of dimensionality. A possible explanation is that common training algorithms for neural networks implicitly perform dimensionality reduction - a process called feature learning. Recent work posited that the effects of feature learning can be elicited from a classical statistical estimator called the average gradient outer product (AGOP). The authors proposed Recursive Feature Machines (RFMs) as an algorithm that explicitly performs feature learning by alternating between (1) reweighting the feature vectors by the AGOP and (2) learning the prediction function in the transformed space. In this work, we develop the first theoretical guarantees for how RFM performs dimensionality reduction by focusing on the class of overparametrized problems arising in sparse linear regression and low-rank matrix recovery. Specifically, we show that RFM restricted to linear models (lin-RFM) generalizes the well-studied Iteratively Reweighted Least Squares (IRLS) algorithm. Our results shed light on the connection between feature learning in neural networks and classical sparse recovery algorithms. In addition, we provide an implementation of lin-RFM that scales to matrices with millions of missing entries. Our implementation is faster than the standard IRLS algorithm as it is SVD-free. It also outperforms deep linear networks for sparse linear regression and low-rank matrix completion.
DSReg: Provably Recovering Individual World Latents without Reconstruction
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Cold Spring Harbor Laboratory（冷泉港实验室）；Brown University（布朗大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ELLIS Institute Tübingen（ELLIS图宾根研究所）
Average Gradient Outer Product in kernel regression provably recovers the central subspace for multi-index models
Department of Mathematics, University of Washington, Seattle, WA 98195（华盛顿大学数学系，华盛顿州西雅图98195）；Wharton Department of Statistics and Data Science, University of Pennsylvania, Philadelphia, PA 19104, USA（宾夕法尼亚大学沃顿统计与数据科学系，美国费城19104）；Department of Mathematics, U. Washington, Seattle, WA 98195（华盛顿大学数学系，华盛顿州西雅图98195）；Department of Electrical & Computer Engineering, University of Washington, Seattle, WA 98195, and Amazon, Inc（华盛顿大学电气与计算机工程系，华盛顿州西雅图98195，亚马逊公司）
Carnegie Mellon University（卡内基梅隆大学）；Courant Institute of Mathematical Sciences, New York University（纽约大学柯朗数学研究所）；University of Southern California（南加州大学）；University of Pennsylvania（宾夕法尼亚大学）
Provably Efficient Risk-Sensitive Offline Reinforcement Learning via Optimized Certainty Equivalents
City University of Hong Kong（香港城市大学）；University of Chicago（芝加哥大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai Jiao Tong University（上海交通大学）
Provably robust blind source separation of linear-quadratic near-separable mixtures
Télécom Paris, Institut Polytechnique de Paris（巴黎电信学院，巴黎综合理工学院）；University of Mons（蒙斯大学）；University of Toulouse, IRIT/INP-ENSEEIHT（图卢兹大学，IRIT/INP-ENSEEIHT）
