Random features is one of the most popular techniques to speed up kernel methods in large-scale problems. Related works have been recognized by the NeurIPS Test-of-Time award in 2017 and the ICML Best Paper Finalist in 2019. The body of work on random features has grown rapidly, and hence it is desirable to have a comprehensive overview on this topic explaining the connections among various algorithms and theoretical results. In this survey, we systematically review the work on random features from the past ten years. First, the motivations, characteristics and contributions of representative random features based algorithms are summarized according to their sampling schemes, learning procedures, variance reduction properties and how they exploit training data. Second, we review theoretical results that center around the following key question: how many random features are needed to ensure a high approximation quality or no loss in the empirical/expected risks of the learned estimator. Third, we provide a comprehensive evaluation of popular random features based algorithms on several large-scale benchmark datasets and discuss their approximation quality and prediction performance for classification. Last, we discuss the relationship between random features and modern over-parameterized deep neural networks (DNNs), including the use of high dimensional random features in the analysis of DNNs as well as the gaps between current theoretical and empirical results. This survey may serve as a gentle introduction to this topic, and as a users' guide for practitioners interested in applying the representative algorithms and understanding theoretical results under various technical assumptions. We hope that this survey will facilitate discussion on the open problems in this topic, and more importantly, shed light on future research directions.
Estimation Beyond Data Reweighting: Kernel Method of Moments
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Weierstrass Institute for Applied Analysis and Stochastics（魏尔斯特拉斯应用分析与随机学研究所）；Eidgenössische Technische Hochschule Zürich（苏黎世联邦理工学院）
Kernel Mean Embedding of Distributions: A Review and Beyond
Mahidol University（玛希隆大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Institute of Statistical Mathematics（统计数学研究所）；Pennsylvania State University（宾夕法尼亚州立大学）
Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents
Center for Intelligent Decision-Making and Machine Learning, School of Management, Xi’an Jiaotong University, Xi’an, China（智能决策与机器学习中心，管理学院，西安交通大学，西安，中国）；Department of Mathematics, The University of Hong Kong, Hongkong, China（数学系，香港大学，香港，中国）
Kernel machines that adapt to GPUs for effective large batch training
Ohio State University（俄亥俄州立大学）
现代机器学习模型通常使用随机梯度下降（SGD）在诸如GPU的大规模并行计算资源上进行训练。增加小批量是利用并行计算能力的一种简单而直接的方法。对于小批量，批量大小的增加会导致训练时间的成比例减少，这种现象称为线性缩放。但是，将批大小增加到超过某个值不会进一步改善训练时间。在本文中，我们开发的第一个分析框架，扩展线性缩放，以匹配并行计算能力的资源。该框架是为一类经典的内核机器设计的。它自动修改标准内核机器以输出数学上等效的预测函数，但允许扩展的线性缩放，即，在给定硬件上实现更高效的并行化和更快的训练时间。由此产生的算法是准确的，原则性和非常快。例如，使用单个Titan Xp GPU，在ImageNet上使用$1.3\times 10^6$数据点和$1000$标签进行训练需要不到一个小时，而较小的数据集（如MNIST）需要几秒钟。由于参数是基于理论界解析地选择的，因此除了选择核和核参数之外，几乎不需要调整，进一步促进了这些方法的实际使用。
Unified Framework for Spectral Dimensionality Reduction, Maximum Variance Unfolding, and Kernel Learning By Semidefinite Programming: Tutorial and Survey
Efficient Global String Kernel with Random Features: Beyond Counting Substructures
IBM Research（IBM研究院）；Carnegie Mellon University（卡内基梅隆大学）；George Mason University（乔治梅森大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）；Zhejiang University（浙江大学）
Beyond State Space Representation: A General Theory for Kernel Packets
School of Data Science, Fudan University（复旦大学数据科学学院）；Wm Michael Barnes ’64 Department of Industrial & Systems Engineering Texas A&M University（德克萨斯大学阿姆斯特朗分校工业与系统工程系）；School of Management, Shenzhen Polytechnic University（深圳职业技术大学管理学院）
