This paper makes a substantial step towards cloning the functionality of black-box models by introducing a Machine learning (ML) architecture named Deep Neural Trees (DNTs). This new architecture can learn to separate different tasks of the black-box model, and clone its task-specific behavior. We propose to train the DNT using an active learning algorithm to obtain faster and more sample-efficient training. In contrast to prior work, we study a complex "victim" black-box model based solely on input-output interactions, while at the same time the attacker and the victim model may have completely different internal architectures. The attacker is a ML based algorithm whereas the victim is a generally unknown module, such as a multi-purpose digital chip, complex analog circuit, mechanical system, software logic or a hybrid of these. The trained DNT module not only can function as the attacked module, but also provides some level of explainability to the cloned model due to the tree-like nature of the proposed architecture.
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；CISPA Helmholtz Center i.G.（CISPA亥姆霍兹信息安全中心）
Machine Learning (ML) models are increasingly deployed in the wild to perform a wide range of tasks. In this work, we ask to what extent can an adversary steal functionality of such "victim" models based solely on blackbox interactions: image in, predictions out. In contrast to prior work, we present an adversary lacking knowledge of train/test data used by the model, its internals, and semantics over model outputs. We formulate model functionality stealing as a two-step approach: (i) querying a set of input images to the blackbox model to obtain predictions; and (ii) training a "knockoff" wit…
In this paper, we present a novel insider attack called Matryoshka, which employs an irrelevant scheduled-to-publish DNN model as a carrier model for covert transmission of multiple secret models which memorize the functionality of private ML data stored in local data centers. Instead of treating the parameters of the carrier model as bit strings and applying conventional steganography, we devise a novel parameter sharing approach which exploits the learning capacity of the carrier model for information hiding. Matryoshka simultaneously achieves: (i) High Capacity -- With almost no utility los…
Georgia Institute of Technology（佐治亚理工学院）；University of Michigan（密歇根大学）
Model Stealing (MS) attacks allow an adversary with black-box access to a Machine Learning model to replicate its functionality, compromising the confidentiality of the model. Such attacks train a clone model by using the predictions of the target model for different inputs. The effectiveness of such attacks relies heavily on the availability of data necessary to query the target model. Existing attacks either assume partial access to the dataset of the target model or availability of an alternate dataset with semantic similarities. This paper proposes MAZE -- a data-free model stealing attack…
Norton Research Group（诺顿研究组）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；INRIA（法国国家信息与自动化研究所）
Many real-world data come in the form of graphs. Graph neural networks (GNNs), a new family of machine learning (ML) models, have been proposed to fully leverage graph data to build powerful applications. In particular, the inductive GNNs, which can generalize to unseen data, become mainstream in this direction. Machine learning models have shown great potential in various tasks and have been deployed in many real-world scenarios. To train a good model, a large amount of data as well as computational resources are needed, leading to valuable intellectual property. Previous research has shown t…
Graduate School of Computer Science, Saarland University（萨尔大学计算机科学研究生院）；Bosch Center for Artificial Intelligence, Robert Bosch GmbH（罗伯特·博世有限公司博世人工智能中心）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）
This paper presents the first model extraction attack against Deep Reinforcement Learning (DRL), which enables an external adversary to precisely recover a black-box DRL model only from its interaction with the environment. Model extraction attacks against supervised Deep Learning models have been widely studied. However, those techniques cannot be applied to the reinforcement learning scenario due to DRL models' high complexity, stochasticity and limited observable information. We propose a novel methodology to overcome the above challenges. The key insight of our approach is that the process…
Efficient Model-Stealing Attacks Against Inductive Graph Neural Networks
IDEAS NCBR；Institute of Fundamental Technological Research, Polish Academy of Sciences（波兰科学院基础技术研究所）；Warsaw University of Technology（华沙理工大学）；CISPA（CISPA亥姆霍兹信息安全中心）；University of Warsaw（华沙大学）
CISPA Helmholtz Center for Information Security（CISPA 亥姆霍兹信息安全中心）；Duke University（杜克大学）
Graph data, such as chemical networks and social networks, may be deemed confidential/private because the data owner often spends lots of resources collecting the data or the data contains sensitive information, e.g., social relationships. Recently, neural networks were extended to graph data, which are known as graph neural networks (GNNs). Due to their superior performance, GNNs have many applications, such as healthcare analytics, recommender systems, and fraud detection. In this work, we propose the first attacks to steal a graph from the outputs of a GNN model that is trained on the graph…
Indraprastha Institute of Information Technology, Delhi（德里英德拉普拉斯塔信息技术学院）；Indian Institute of Technology, Kharagpur（印度理工学院克勒格布尔分校）；Institute for Systems Studies and Analyses（系统研究与分析研究所）；Aurel Vlaicu University of Arad（阿拉德奥雷尔·弗拉伊库大学）
Deep learning is gaining importance in many applications. However, Neural Networks face several security and privacy threats. This is particularly significant in the scenario where Cloud infrastructures deploy a service with Neural Network model at the back end. Here, an adversary can extract the Neural Network parameters, infer the regularization hyperparameter, identify if a data point was part of the training data, and generate effective transferable adversarial examples to evade classifiers. This paper shows how a Neural Network model is susceptible to timing side channel attack. In this p…
IBM Research（IBM研究院）；Bar Ilan University（巴伊兰大学）；Tel Aviv University（特拉维夫大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Deep neural networks work well at approximating complicated functions when provided with data and trained by gradient descent methods. At the same time, there is a vast amount of existing functions that programmatically solve different tasks in a precise manner eliminating the need for training. In many cases, it is possible to decompose a task to a series of functions, of which for some we may prefer to use a neural network to learn the functionality, while for others the preferred method would be to use existing black-box functions. We propose a method for end-to-end training of a base neura…
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）
The wide application of deep learning techniques is boosting the regulation of deep learning models, especially deep neural networks (DNN), as commercial products. A necessary prerequisite for such regulations is identifying the owner of deep neural networks, which is usually done through the watermark. Current DNN watermarking schemes, particularly white-box ones, are uniformly fragile against a family of functionality equivalence attacks, especially the neuron permutation. This operation can effortlessly invalidate the ownership proof and escape copyright regulations. To enhance the robustne…
Adversaries may look to steal or attack black-box NLP systems, either for financial gain or to exploit model errors. One setting of particular interest is machine translation (MT), where models have high commercial value and errors can be costly. We investigate possible exploits of black-box MT systems and explore a preliminary defense against such threats. We first show that MT systems can be stolen by querying them with monolingual sentences and training models to imitate their outputs. Using simulated experiments, we demonstrate that MT model stealing is possible even when imitation models …
GLiRA: Black-Box Membership Inference Attack via Knowledge Distillation
Artificial Intelligence Research Institute（人工智能研究所）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；ISP RAS Research Center for Trusted Artificial Intelligence（俄罗斯科学院信息学研究所可信人工智能研究中心）
In this paper, we present a black-box attack against API call based machine learning malware classifiers, focusing on generating adversarial sequences combining API calls and static features (e.g., printable strings) that will be misclassified by the classifier without affecting the malware functionality. We show that this attack is effective against many classifiers due to the transferability principle between RNN variants, feed forward DNNs, and traditional machine learning classifiers such as SVM. We also implement GADGET, a software framework to convert any malware binary to a binary undet…
Protecting Deep Learning Model Copyrights with Adversarial Example-Free Reuse Detection
School of Mathematical Sciences, Peking University, China（北京大学数学科学学院）；School of Computer Science, University of Bristol, UK（英国布里斯托大学计算机科学学院）；School of Control Science and Engineering, Zhejiang University, China（浙江大学控制科学与工程学院）
提出 NFARD，利用神经元功能与正常样本检测 DNN 复用关系，支持白盒、黑盒及异构场景，并构建 Reuse Zoo，检测 F1 最高达 1.0。
MOVE: Effective and Harmless Ownership Verification via Embedded External Features
Zhejiang University（浙江大学）；Tsinghua University（清华大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Nanyang Technological University（南洋理工大学）；ByteDance Inc（字节跳动公司）；Peng Cheng Laboratory（鹏城实验室）；Sun Yat-sen University（中山大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新区（滨江）区块链与数据安全研究院）
Copyright protection for deep neural networks (DNNs) is an urgent need for AI corporations. To trace illegally distributed model copies, DNN watermarking is an emerging technique for embedding and verifying secret identity messages in the prediction behaviors or the model internals. Sacrificing less functionality and involving more knowledge about the target DNN, the latter branch called \textit{white-box DNN watermarking} is believed to be accurate, credible and secure against most known watermark removal attacks, with emerging research efforts in both the academy and the industry. In this pa…
In this paper, we present a generic, query-efficient black-box attack against API call-based machine learning malware classifiers. We generate adversarial examples by modifying the malware's API call sequences and non-sequential features (printable strings), and these adversarial examples will be misclassified by the target malware classifier without affecting the malware's functionality. In contrast to previous studies, our attack minimizes the number of malware classifier queries required. In addition, in our attack, the attacker must only know the class predicted by the malware classifier; …
CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全研究中心）；DeepMind（深度思维）
In this work, we investigate the problem of revealing the functionality of a black-box agent. Notably, we are interested in the interpretable and formal description of the behavior of such an agent. Ideally, this description would take the form of a program written in a high-level language. This task is also known as reverse engineering and plays a pivotal role in software engineering, computer security, but also most recently in interpretability. In contrast to prior work, we do not rely on privileged information on the black box, but rather investigate the problem under a weaker assumption o…
University of Bucharest（布加勒斯特大学）；University Politehnica of Bucharest（布加勒斯特理工大学）
We study the task of replicating the functionality of black-box neural models, for which we only know the output class probabilities provided for a set of input images. We assume back-propagation through the black-box model is not possible and its training images are not available, e.g. the model could be exposed only through an API. In this context, we present a teacher-student framework that can distill the black-box (teacher) model into a student model with minimal accuracy loss. To generate useful data samples for training the student, our framework (i) learns to generate images on a proxy…
BridgePure: Limited Protection Leakage Can Break Black-Box Data Protection
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）
Technische Universität Wien（维也纳工业大学）；New York University Abu Dhabi（纽约大学阿布扎比分校）
The advancements of deep neural networks (DNNs) have led to their deployment in diverse settings, including safety and security-critical applications. As a result, the characteristics of these models have become sensitive intellectual properties that require protection from malicious users. Extracting the architecture of a DNN through leaky side-channels (e.g., memory access) allows adversaries to (i) clone the model, and (ii) craft adversarial attacks. DNN obfuscation thwarts side-channel-based architecture stealing (SCAS) attacks by altering the run-time traces of a given DNN while preservin…
University College London（伦敦大学学院）；University of Warwick（华威大学）；University of Oxford（牛津大学）
Predictive modelling and supervised learning are central to modern data science. With predictions from an ever-expanding number of supervised black-box strategies - e.g., kernel methods, random forests, deep learning aka neural networks - being employed as a basis for decision making processes, it is crucial to understand the statistical uncertainty associated with these predictions. As a general means to approach the issue, we present an overarching framework for black-box prediction strategies that not only predict the target but also their own predictions' uncertainty. Moreover, the framewo…
