Data sampling is an effective method to improve the training speed of neural networks, with recent results demonstrating that it can even break the neural scaling laws. These results critically rely on high-quality scores to estimate the importance of an input to the network. We observe that there are two dominant strategies: static sampling, where the scores are determined before training, and dynamic sampling, where the scores can depend on the model weights. Static algorithms are computationally inexpensive but less effective than their dynamic counterparts, which can cause end-to-end slowdown due to their need to explicitly compute losses. To address this problem, we propose a novel sampling distribution based on nonparametric kernel regression that learns an effective importance score as the neural network trains. However, nonparametric regression models are too computationally expensive to accelerate end-to-end training. Therefore, we develop an efficient sketch-based approximation to the Nadaraya-Watson estimator. Using recent techniques from high-dimensional statistics and randomized algorithms, we prove that our Nadaraya-Watson sketch approximates the estimator with exponential convergence guarantees. Our sampling algorithm outperforms the baseline in terms of wall-clock time and accuracy on four datasets.
Optimal Sampling Density for Nonparametric Regression
Berlin Institute of Technology(柏林工业大学);BIFOLD-Berlin Institute for the Foundations of Learning and Data(BIFOLD-柏林学习与数据基础研究所);Korea University(高丽大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);RIKEN AIP(理化学研究所人工智能研究中心)
Beyond Static Bias: Adaptive Multi-Fidelity Bandits with Improving Proxies
Department of Industrial and Systems Engineering, University of Houston(休斯敦大学工业与系统工程系);School of Electrical Engineering and Computer Science, Oregon State University(俄勒冈州立大学电气工程与计算机科学学院)
TG-NAS: Generalizable Zero-Cost Proxies with Operator Description Embedding and Graph Learning for Efficient Neural Architecture Search
Department of EECS, University of California, Irvine(电子工程与计算机科学系,加州大学伊市分校);Department of Computer Science, University of California, San Diego(计算机科学系,加州大学圣地亚哥分校)
Southwestern University of Finance and Economics(西南财经大学);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);Wuhan University(武汉大学)
Conditional Diffusion for Nonparametric Instrumental Variable Quantile Regression
Institute of Data Science and Statistics, Shanghai University of Finance and Economics(上海财经大学数据科学与统计研究院);School of Mathematics and Statistics, Wuhan University(武汉大学数学与统计学院);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院)