A plethora of recent research has focused on improving the memory footprint and inference speed of deep networks by reducing the complexity of (i) numerical representations (for example, by deterministic or stochastic quantization) and (ii) arithmetic operations (for example, by binarization of weights).
We propose a stochastic binarization scheme for deep networks that allows for efficient inference on hardware by restricting itself to additions of small integers and fixed shifts. Unlike previous approaches, the underlying randomized approximation is progressive, thus permitting an adaptive control of the accuracy of each operation at run-time. In a low-precision setting, we match the accuracy of previous binarized approaches. Our representation is unbiased - it approaches continuous computation with increasing sample size. In a high-precision regime, the computational costs are competitive with previous quantization schemes. Progressive stochastic binarization also permits localized, dynamic accuracy control within a single network, thereby providing a new tool for adaptively focusing computational attention.
We evaluate our method on networks of various architectures, already pretrained on ImageNet. With representational costs comparable to previous schemes, we obtain accuracies close to the original floating point implementation. This includes pruned networks, except the known special case of certain types of separated convolutions. By focusing computational attention using progressive sampling, we reduce inference costs on ImageNet further by a factor of up to 33% (before network pruning).
Peng Cheng Laboratory（鹏城实验室）；Fudan University（复旦大学）；Institute of Microelectronics, Chinese Academy of Sciences（中国科学院微电子研究所）；Politecnico di Milano（米兰理工大学）
KAUST Center of Excellence for Generative AI（卡斯土尼亚大学生成人工智能卓越中心）；Shenzhen University School of Mathematical Sciences（深圳大学数学科学学院）；King Abdullah University of Science and Technology（国王阿卜杜勒-阿齐兹大学科学与技术大学）
Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Korea University（高丽大学）
Learning stochastic object models from medical imaging measurements by use of advanced ambient generative adversarial networks
University of California Santa Barbara（加州大学圣巴巴拉分校）；Washington University in St. Louis（圣路易斯华盛顿大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Cancer Center at Illinois（伊利诺伊癌症中心）
