arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1708.00630cs.LGcs.AIcs.NE
收藏

ProjectionNet:使用神经投影学习高效的设备上深度网络

ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections

Google Research(谷歌研究院)

Sujith Ravi

摘要:

深度神经网络在与视觉识别和语言理解任务相关的应用中已经变得无处不在。然而,在像移动电话或智能手表这样的设备上使用典型的神经网络通常是禁止的,因为模型大小很大,并且不能适应这些设备上可用的有限存储器。虽然这些设备可以利用在具有CPU或GPU的高性能数据中心上运行的机器学习模型,但这对于许多应用来说是不可行的,因为数据可能是隐私敏感的,并且需要直接在设备上执行推理。我们介绍了一种新的架构,用于训练紧凑的神经网络使用联合优化框架。其核心在于一个新的目标,即使用两种不同类型的网络进行联合训练-一个完整的训练神经网络(使用前馈NN或LSTM RNN等现有架构)与一个更简单的“投影”网络相结合,该网络利用随机投影将输入或中间表示转换为比特。更简单的网络在位空间中编码轻量级和有效的计算操作,内存占用量低。这两个网络使用反向传播进行联合训练,其中投影网络从完整网络中学习,类似于学徒学习。一旦经过训练,较小的网络可以直接用于推理,内存和计算成本较低。我们证明了新方法的有效性,在显着缩小不同类型的神经网络的内存需求,同时保持良好的准确性视觉识别和文本分类任务。我们还研究了“解决一个给定的任务需要多少个神经比特”的问题。“使用新的框架,并显示实证结果对比模型预测能力(比特)与几个数据集的准确性。

英文摘要:

Deep neural networks have become ubiquitous for applications related to visual recognition and language understanding tasks. However, it is often prohibitive to use typical neural networks on devices like mobile phones or smart watches since the model sizes are huge and cannot fit in the limited memory available on such devices. While these devices could make use of machine learning models running on high-performance data centers with CPUs or GPUs, this is not feasible for many applications because data can be privacy sensitive and inference needs to be performed directly "on" device. We introduce a new architecture for training compact neural networks using a joint optimization framework. At its core lies a novel objective that jointly trains using two different types of networks--a full trainer neural network (using existing architectures like Feed-forward NNs or LSTM RNNs) combined with a simpler "projection" network that leverages random projections to transform inputs or intermediate representations into bits. The simpler network encodes lightweight and efficient-to-compute operations in bit space with a low memory footprint. The two networks are trained jointly using backpropagation, where the projection network learns from the full network similar to apprenticeship learning. Once trained, the smaller network can be used directly for inference at low memory and computation cost. We demonstrate the effectiveness of the new approach at significantly shrinking the memory requirements of different types of neural networks while preserving good accuracy on visual recognition and text classification tasks. We also study the question "how many neural bits are required to solve a given task?" using the new framework and show empirical results contrasting model predictive capacity (in bits) versus accuracy on several datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑