基于监督学习的深度神经网络算法选择
Supervised Learning Based Algorithm Selection for Deep Neural Networks
摘要：
许多最近的深度学习平台依赖于第三方库（如cuBLAS）来利用现代硬件加速器（如GPU）的计算能力。然而，我们观察到，它们可能会达到次优性能，因为库函数没有得到适当的使用。在本文中，我们的目标是优化一个矩阵与另一个矩阵的转置相乘的操作（以下称为NT操作），这占了全连接深度神经网络训练时间的一半左右。我们提出了一种基于监督学习的算法选择方法MTNN，而不是直接调用库函数，该方法使用梯度提升的决策树来智能地从两个备选NT实现中选择一个：（1）调用cuBLAS库函数;（2）调用我们提出的算法TNN，该算法使用有效的错位矩阵转置。我们评估了MTNN在两个现代GPU上的性能：NVIDIA GTX 1080和NVIDIA Titan X Pascal。MTNN可以达到96%的预测精度与非常低的计算开销，这导致在一系列NT操作的平均性能提高54%.为了进一步评估MTNN对深度神经网络训练过程的影响，我们将MTNN集成到流行的深度学习平台Caffe中。实验结果表明，改进后的Caffe算法的性能平均比原算法提高了28%. MTNN和修改后的Caffe都是开源的。
英文摘要：
Many recent deep learning platforms rely on third-party libraries (such as cuBLAS) to utilize the computing power of modern hardware accelerators (such as GPUs). However, we observe that they may achieve suboptimal performance because the library functions are not used appropriately. In this paper, we target at optimizing the operations of multiplying a matrix with the transpose of another matrix (referred to as NT operation hereafter), which contribute about half of the training time of fully connected deep neural networks. Rather than directly calling the library function, we propose a supervised learning based algorithm selection approach named MTNN, which uses a gradient boosted decision tree to select one from two alternative NT implementations intelligently: (1) calling the cuBLAS library function; (2) calling our proposed algorithm TNN that uses an efficient out-of-place matrix transpose. We evaluate the performance of MTNN on two modern GPUs: NVIDIA GTX 1080 and NVIDIA Titan X Pascal. MTNN can achieve 96\% of prediction accuracy with very low computational overhead, which results in an average of 54\% performance improvement on a range of NT operations. To further evaluate the impact of MTNN on the training process of deep neural networks, we have integrated MTNN into a popular deep learning platform Caffe. Our experimental results show that the revised Caffe can outperform the original one by an average of 28\%. Both MTNN and the revised Caffe are open-source.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
