Deep neural networks have become ubiquitous for applications related to visual recognition and language understanding tasks. However, it is often prohibitive to use typical neural networks on devices like mobile phones or smart watches since the model sizes are huge and cannot fit in the limited memory available on such devices. While these devices could make use of machine learning models running on high-performance data centers with CPUs or GPUs, this is not feasible for many applications because data can be privacy sensitive and inference needs to be performed directly "on" device.
We introduce a new architecture for training compact neural networks using a joint optimization framework. At its core lies a novel objective that jointly trains using two different types of networks--a full trainer neural network (using existing architectures like Feed-forward NNs or LSTM RNNs) combined with a simpler "projection" network that leverages random projections to transform inputs or intermediate representations into bits. The simpler network encodes lightweight and efficient-to-compute operations in bit space with a low memory footprint. The two networks are trained jointly using backpropagation, where the projection network learns from the full network similar to apprenticeship learning. Once trained, the smaller network can be used directly for inference at low memory and computation cost. We demonstrate the effectiveness of the new approach at significantly shrinking the memory requirements of different types of neural networks while preserving good accuracy on visual recognition and text classification tasks. We also study the question "how many neural bits are required to solve a given task?" using the new framework and show empirical results contrasting model predictive capacity (in bits) versus accuracy on several datasets.
Random Projections through multiple optical scattering: Approximating kernels at the speed of light
Laboratoire de Physique Statistique, CNRS UMR 8550 & École Normale Supérieure(统计物理实验室,CNRS UMR 8550 & 巴黎高等师范学院);Institut Langevin, ESPCI and CNRS UMR 7587(朗之万研究所,ESPCI 与 CNRS UMR 7587);Paris Diderot University, Sorbonne Paris Cité(巴黎狄德罗大学,索邦巴黎西岱);ENSTA Bretagne and Lab-STICC UMR 6285(ENSTA 布列塔尼与 Lab-STICC UMR 6285);Laboratoire Kastler Brossel, CNRS UMR 8552 & École Normale Supérieure(卡斯特勒-布罗塞尔实验室,CNRS UMR 8552 & 巴黎高等师范学院);Sorbonne Universités, Université Pierre et Marie Curie Paris 06(索邦大学群,皮埃尔和玛丽·居里大学(巴黎第六大学));PSL Research University(巴黎文理研究大学)
HPD: Hybrid Projection Decomposition for Robust State Space Models on Analog CIM Hardware
School of Integrated Circuit Science and Engineering(集成电路科学与工程学院);Beihang University(北航);Department of Electrical and Electronic Engineering(电子电气工程系);The University of Hong Kong(香港大学);Zhicun Research Lab(智存研究实验室)