低秩梯度近似用于深度神经网络的存储器有效的设备上训练
Low-rank Gradient Approximation For Memory-Efficient On-device Training of Deep Neural Network
摘要：
在移动设备上训练机器学习模型有可能提高模型的隐私性和准确性。然而，实现这一目标的主要障碍之一是移动设备的内存限制。减少训练内存使具有高维权重矩阵的模型（如自动语音识别（ASR）模型）能够在设备上进行训练。在本文中，我们提出使用低秩参数化来近似深度神经网络的梯度矩阵，作为节省训练内存的一种途径。低秩梯度近似使更高级的内存密集型优化技术能够在设备上运行。我们的实验结果表明，我们可以减少约33.0%的亚当优化的训练记忆。它使用了与动量优化相当的内存，并在ASR个性化任务中实现了4.5%的相对较低的字错误率。
英文摘要：
Training machine learning models on mobile devices has the potential of improving both privacy and accuracy of the models. However, one of the major obstacles to achieving this goal is the memory limitation of mobile devices. Reducing training memory enables models with high-dimensional weight matrices, like automatic speech recognition (ASR) models, to be trained on-device. In this paper, we propose approximating the gradient matrices of deep neural networks using a low-rank parameterization as an avenue to save training memory. The low-rank gradient approximation enables more advanced, memory-intensive optimization techniques to be run on device. Our experimental results show that we can reduce the training memory by about 33.0% for Adam optimization. It uses comparable memory to momentum optimization and achieves a 4.5% relative lower word error rate on an ASR personalization task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
