Outsourcing deep neural networks (DNNs) inference tasks to an untrusted cloud raises data privacy and integrity concerns. While there are many techniques to ensure privacy and integrity for polynomial-based computations, DNNs involve non-polynomial computations. To address these challenges, several privacy-preserving and verifiable inference techniques have been proposed based on replacing the non-polynomial activation functions such as the rectified linear unit (ReLU) function with polynomial activation functions. Such techniques usually require polynomials with integer coefficients or polynomials over finite fields. Motivated by such requirements, several works proposed replacing the ReLU function with the square function. In this work, we empirically show that the square function is not the best degree-2 polynomial that can replace the ReLU function even when restricting the polynomials to have integer coefficients. We instead propose a degree-2 polynomial activation function with a first order term and empirically show that it can lead to much better models. Our experiments on the CIFAR and Tiny ImageNet datasets on various architectures such as VGG-16 show that our proposed function improves the test accuracy by up to 10.4% compared to the square function.
Privacy concerns in client-server machine learning have given rise to private inference (PI), where neural inference occurs directly on encrypted inputs. PI protects clients' personal data and the server's intellectual property. A common practice in PI is to use garbled circuits to compute nonlinear functions privately, namely ReLUs. However, garbled circuits suffer from high storage, bandwidth, and latency costs. To mitigate these issues, PI-friendly polynomial activation functions have been employed to replace ReLU. In this work, we ask: Is it feasible to substitute all ReLUs with low-degree…
The rapid growth and deployment of deep learning (DL) has witnessed emerging privacy and security concerns. To mitigate these issues, secure multi-party computation (MPC) has been discussed, to enable the privacy-preserving DL computation. In practice, they often come at very high computation and communication overhead, and potentially prohibit their popularity in large scale systems. Two orthogonal research trends have attracted enormous interests in addressing the energy efficiency in secure deep learning, i.e., overhead reduction of MPC comparison protocol, and hardware acceleration. Howeve…
Indian Institute of Engineering Science and Technology, Shibpur（希布尔印度工程科学与技术学院）；University of New South Wales（新南威尔士大学）
Recently collaborative learning is widely applied to model sensitive data generated in Industrial IoT (IIoT). It enables a large number of devices to collectively train a global model by collaborating with a server while keeping the datasets on their respective premises. However, existing approaches are limited by high overheads and may also suffer from falsified aggregated results returned by a malicious server. Hence, we propose a Scalable, Privacy-preserving and veRIfiable collaboraTive lEarning (SPRITE) algorithm to train linear and logistic regression models for IIoT. We aim to reduce bur…
ZTFed-MAS2S: A Zero-Trust Federated Learning Framework with Verifiable Privacy and Trust-Aware Aggregation for Wind Power Data Imputation
School of Electrical Engineering, Northeast Electric Power University（电气工程学院，东北电力大学）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（人工智能与自动化学院，华中科技大学）；State Grid Xiamen Electric Power Supply Company（国家电网厦门供电有限公司）；Department of Electrical Engineering, City University of Hong Kong（电气工程系，香港城市大学）
The recent rise of privacy concerns has led researchers to devise methods for private neural inference -- where inferences are made directly on encrypted data, never seeing inputs. The primary challenge facing private inference is that computing on encrypted data levies an impractically-high latency penalty, stemming mostly from non-linear operators like ReLU. Enabling practical and private inference requires new optimization methods that minimize network ReLU counts while preserving accuracy. This paper proposes DeepReDuce: a set of optimizations for the judicious removal of ReLUs to reduce p…
Privacy-Preserving and Verifiable Approximate Distributed Coded Computing
Galician Research and Development Center in Advanced Telecommunications (GRADIANT)（加利西亚先进电信研究与发展中心（GRADIANT））；atlanTTic, Information & Computing Lab (ICLAB), Telecommunication Engineering School, Universidade de Vigo（atlanTTic，信息与计算实验室（ICLAB），电信工程学校，维戈大学）
University of Connecticut（康涅狄格大学）；Northeastern University（东北大学）；Lehigh University（理海大学）；Duke University（杜克大学）；University of Rochester（罗切斯特大学）
The proliferation of deep learning (DL) has led to the emergence of privacy and security concerns. To address these issues, secure Two-party computation (2PC) has been proposed as a means of enabling privacy-preserving DL computation. However, in practice, 2PC methods often incur high computation and communication overhead, which can impede their use in large-scale systems. To address this challenge, we introduce RRNet, a systematic framework that aims to jointly reduce the overhead of MPC comparison protocols and accelerate computation through hardware acceleration. Our approach integrates th…
New York University（纽约大学）；University of California San Diego（加州大学圣地亚哥分校）
The emergence of deep learning has been accompanied by privacy concerns surrounding users' data and service providers' models. We focus on private inference (PI), where the goal is to perform inference on a user's data sample using a service provider's model. Existing PI methods for deep networks enable cryptographically secure inference with little drop in functionality; however, they incur severe latency costs, primarily caused by non-linear network operations (such as ReLUs). This paper presents Sphynx, a ReLU-efficient network design method based on micro-search strategies for convolutiona…
ZKP-FedEval: Verifiable and Privacy-Preserving Federated Evaluation using Zero-Knowledge Proofs
Department of Multidisciplinary Engineering, Texas A&M University（多学科工程系，德克萨斯A&M大学）；Department of Computer Science, Ho Technical University（计算机科学系，霍技术大学）；Department of Computer Engineering, Kwame Nkrumah University of Science and Technology (KNUST)（计算机工程系，库马西科学与技术大学（KNUST））
Learning from data owned by several parties, as in federated learning, raises challenges regarding the privacy guarantees provided to participants and the correctness of the computation in the presence of malicious parties. We tackle these challenges in the context of distributed averaging, an essential building block of federated learning algorithms. Our first contribution is a scalable protocol in which participants exchange correlated Gaussian noise along the edges of a network graph, complemented by independent noise added by each party. We analyze the differential privacy guarantees of ou…
AutoReP: Automatic ReLU Replacement for Fast Private Network Inference
Northeastern University（东北大学）；Duke University（杜克大学）；Walmart Global Tech（沃尔玛全球科技）；University of Connecticut（康涅狄格大学）；Pacific Northwest National Laboratory（太平洋西北国家实验室）；University of Rochester（罗切斯特大学）；North Carolina State University（北卡罗来纳州立大学）
Columbia University（哥伦比亚大学）；ENS（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL University（巴黎文理研究大学）；INRIA（法国国家信息与自动化研究所）；Arkhn
We propose AriaNN, a low-interaction privacy-preserving framework for private neural network training and inference on sensitive data. Our semi-honest 2-party computation protocol (with a trusted dealer) leverages function secret sharing, a recent lightweight cryptographic protocol that allows us to achieve an efficient online phase. We design optimized primitives for the building blocks of neural networks such as ReLU, MaxPool and BatchNorm. For instance, we perform private comparison for ReLU operations with a single message of the size of the input during the online phase, and with preproce…
PrivEraserVerify: Efficient, Private, and Verifiable Federated Unlearning
Department of Electronics and Communication Engineering（电子与通信工程系）；Department of Mathematics and Computer Science（数学与计算机科学系）；Khulna University of Engineering & Technology（库尔纳工程与技术大学）；Hobart and William Smith Colleges（霍巴特和威廉·斯密斯学院）；Department of Electrical and Electronic Engineering（电气与电子工程系）
Contact Tracing has been used to identify people who were in close proximity to those infected with SARS-Cov2 coronavirus. A number of digital contract tracing applications have been introduced to facilitate or complement physical contact tracing. However, there are a number of privacy issues in the implementation of contract tracing applications, which make people reluctant to install or update their infection status on these applications. In this concept paper, we present ideas from Graph Neural Networks and explainability, that could improve trust in these applications, and encourage adopti…
New York University Tandon School of Engineering（纽约大学坦登工程学院）
Private inference (PI) enables inference directly on cryptographically secure data.While promising to address many privacy issues, it has seen limited use due to extreme runtimes. Unlike plaintext inference, where latency is dominated by FLOPs, in PI non-linear functions (namely ReLU) are the bottleneck. Thus, practical PI demands novel ReLU-aware optimizations. To reduce PI latency we propose a gradient-based algorithm that selectively linearizes ReLUs while maintaining prediction accuracy. We evaluate our algorithm on several standard PI benchmarks. The results demonstrate up to $4.25\%$ mor…
A Survey of Zero-Knowledge Proof Based Verifiable Machine Learning
College of Electronics and Information Engineering, Shenzhen University（深圳大学电子与信息工程学院）；School of Computer Science and Cyber Engineering, Guangzhou University（广州大学计算机科学与网络工程学院）；School of Electronic and Information Engineering, South China University of Technology（华南理工大学电子与信息学院）；State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学网络与交换技术国家重点实验室）；School of Electronic and Optical Engineering, Nanjing University of Science and Technology（南京理工大学电子工程与光电技术学院）
Survey of Privacy-Preserving Collaborative Filtering
Missouri University of Science and Technology（密苏里科技大学）；University of Missouri-Columbia（密苏里大学哥伦比亚分校）；Applied Computational Intelligence Laboratory, Missouri University of Science and Technology（密苏里科技大学应用计算智能实验室）
Because learning sometimes involves sensitive data, machine learning algorithms have been extended to offer privacy for training data. In practice, this has been mostly an afterthought, with privacy-preserving models obtained by re-running training with a different optimizer, but using the model architectures that already performed well in a non-privacy-preserving setting. This approach leads to less than ideal privacy/utility tradeoffs, as we show here. Instead, we propose that model architectures are chosen ab initio explicitly for privacy-preserving training. To provide guarantees under the…
NYU Abu Dhabi（纽约大学阿布扎比分校）；Queen’s University Belfast（贝尔法斯特女王大学）
Machine Learning (ML) architectures have been applied to several applications that involve sensitive data, where a guarantee of users' data privacy is required. Differentially Private Stochastic Gradient Descent (DPSGD) is the state-of-the-art method to train privacy-preserving models. However, DPSGD comes at a considerable accuracy loss leading to sub-optimal privacy/utility trade-offs. Towards investigating new ground for better privacy-utility trade-off, this work questions; (i) if models' hyperparameters have any inherent impact on ML models' privacy-preserving properties, and (ii) if mode…
A common privacy issue in traditional machine learning is that data needs to be disclosed for the training procedures. In situations with highly sensitive data such as healthcare records, accessing this information is challenging and often prohibited. Luckily, privacy-preserving technologies have been developed to overcome this hurdle by distributing the computation of the training and ensuring the data privacy to their owners. The distribution of the computation to multiple participating entities introduces new privacy complications and risks. In this paper, we present a privacy-preserving de…
