Deep Neural Networks (DNNs) are a revolutionary force in the ongoing information revolution, and yet their intrinsic properties remain a mystery. In particular, it is widely known that DNNs are highly sensitive to noise, whether adversarial or random. This poses a fundamental challenge for hardware implementations of DNNs, and for their deployment in critical applications such as autonomous driving. In this paper we construct robust DNNs via error correcting codes. By our approach, either the data or internal layers of the DNN are coded with error correcting codes, and successful computation under noise is guaranteed. Since DNNs can be seen as a layered concatenation of classification tasks, our research begins with the core task of classifying noisy coded inputs, and progresses towards robust DNNs. We focus on binary data and linear codes. Our main result is that the prevalent parity code can guarantee robustness for a large family of DNNs, which includes the recently popularized binarized neural networks. Further, we show that the coded classification problem has a deep connection to Fourier analysis of Boolean functions. In contrast to existing solutions in the literature, our results do not rely on altering the training process of the DNN, and provide mathematically rigorous guarantees rather than experimental evidence.
Coded Robust Aggregation for Distributed Learning under Byzantine Attacks
Division of Information Science and Engineering, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology(信息科学与工程系,电气工程与计算机科学学院,皇家理工学院)
Mallinckrodt Institute of Radiology, Washington University School of Medicine(华盛顿大学医学院马林克罗特放射学研究所);School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);Institute for Informatics, Data Science and Biostatistics, Washington University School of Medicine(华盛顿大学医学院信息学、数据科学与生物统计学研究所)
Speeding Up Distributed Machine Learning Using Codes
School of Electrical Engineering, KAIST(韩国科学技术院电气工程学院);Department of Electrical Engineering and Computer Sciences, University of California, Berkeley(加利福尼亚大学伯克利分校电气工程与计算机科学系);Department of Electrical and Computer Engineering, University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校电气与计算机工程系);Department of Electrical and Computer Engineering, University of Wisconsin-Madison(威斯康星大学麦迪逊分校电气与计算机工程系);Fellow, IEEE(IEEE会士(注:此为职称,非机构,按要求不提取))