arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.13583cs.LG
收藏

基于高效非参数代理的深度学习自适应采样

Adaptive Sampling for Deep Learning via Efficient Nonparametric Proxies

Rice University(莱斯大学);ThirdAI Corp(ThirdAI公司)

Shabnam Daghaghi, Benjamin Coleman, Benito Geordie, Anshumali Shrivastava

AI总结:

针对深度学习动态采样因显式计算损失导致端到端减速的问题,提出基于非参数核回归草图近似 Nadaraya-Watson 估计量的高效采样算法,在保证指数收敛的同时显著提升了训练速度与准确率。

摘要:

数据采样是提升神经网络训练速度的有效方法,近期的研究结果表明它甚至能够打破神经缩放定律。这些结果关键依赖于高质量的分数来估计输入对网络的重要性。我们观察到存在两种主要策略:静态采样,其分数在训练前确定;动态采样,其分数可依赖于模型权重。静态算法计算成本低,但效果不如动态算法,而动态算法由于需要显式计算损失,会导致端到端的减速。为了解决这个问题,我们提出了一种基于非参数核回归的新型采样分布,它能在神经网络训练时学习有效的重要性分数。然而,非参数回归模型的计算成本过高,无法加速端到端训练。因此,我们开发了一种高效的基于草图(sketch)的近似方法来替代 Nadaraya-Watson 估计量。利用高维统计和随机算法的最新技术,我们证明了我们的 Nadaraya-Watson 草图能以指数级收敛保证来近似该估计量。在四个数据集上,我们的采样算法在挂钟时间和准确率方面均优于基线方法。

英文摘要:

Data sampling is an effective method to improve the training speed of neural networks, with recent results demonstrating that it can even break the neural scaling laws. These results critically rely on high-quality scores to estimate the importance of an input to the network. We observe that there are two dominant strategies: static sampling, where the scores are determined before training, and dynamic sampling, where the scores can depend on the model weights. Static algorithms are computationally inexpensive but less effective than their dynamic counterparts, which can cause end-to-end slowdown due to their need to explicitly compute losses. To address this problem, we propose a novel sampling distribution based on nonparametric kernel regression that learns an effective importance score as the neural network trains. However, nonparametric regression models are too computationally expensive to accelerate end-to-end training. Therefore, we develop an efficient sketch-based approximation to the Nadaraya-Watson estimator. Using recent techniques from high-dimensional statistics and randomized algorithms, we prove that our Nadaraya-Watson sketch approximates the estimator with exponential convergence guarantees. Our sampling algorithm outperforms the baseline in terms of wall-clock time and accuracy on four datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑