This paper presents the first model extraction attack against Deep Reinforcement Learning (DRL), which enables an external adversary to precisely recover a black-box DRL model only from its interaction with the environment. Model extraction attacks against supervised Deep Learning models have been widely studied. However, those techniques cannot be applied to the reinforcement learning scenario due to DRL models' high complexity, stochasticity and limited observable information. We propose a novel methodology to overcome the above challenges. The key insight of our approach is that the process of DRL model extraction is equivalent to imitation learning, a well-established solution to learn sequential decision-making policies. Based on this observation, our methodology first builds a classifier to reveal the training algorithm family of the targeted black-box DRL model only based on its predicted actions, and then leverages state-of-the-art imitation learning techniques to replicate the model from the identified algorithm family. Experimental results indicate that our methodology can effectively recover the DRL models with high fidelity and accuracy. We also demonstrate two use cases to show that our model extraction attack can (1) significantly improve the success rate of adversarial attacks, and (2) steal DRL models stealthily even they are protected by DNN watermarks. These pose a severe threat to the intellectual property and privacy protection of DRL applications.
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；CISPA Helmholtz Center i.G.（CISPA亥姆霍兹信息安全中心）
Machine Learning (ML) models are increasingly deployed in the wild to perform a wide range of tasks. In this work, we ask to what extent can an adversary steal functionality of such "victim" models based solely on blackbox interactions: image in, predictions out. In contrast to prior work, we present an adversary lacking knowledge of train/test data used by the model, its internals, and semantics over model outputs. We formulate model functionality stealing as a two-step approach: (i) querying a set of input images to the blackbox model to obtain predictions; and (ii) training a "knockoff" wit…
Graduate School of Computer Science, Saarland University（萨尔大学计算机科学研究生院）；Bosch Center for Artificial Intelligence, Robert Bosch GmbH（罗伯特·博世有限公司博世人工智能中心）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）
Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learning
Department of Computer Science, University of Texas at Austin, Austin, TX, USA（德克萨斯大学奥斯汀分校计算机科学系）；Department of Aerospace Engineering and Engineering Mechanics, University of Texas at Austin, Austin, TX, USA（德克萨斯大学奥斯汀分校航空航天工程与工程力学系）
University of Agder（阿格德大学）；SpareBank 1 SR-Bank（SpareBank 1 SR银行）
Stock portfolio optimization is the process of continuous reallocation of funds to a selection of stocks. This is a particularly well-suited problem for reinforcement learning, as daily rewards are compounding and objective functions may include more than just profit, e.g., risk and sustainability. We developed a novel utility function with the Sharpe ratio representing risk and the environmental, social, and governance score (ESG) representing sustainability. We show that a state-of-the-art policy gradient method - multi-agent deep deterministic policy gradients (MADDPG) - fails to find the o…
Existing recommendation algorithms mostly focus on optimizing traditional recommendation measures, such as the accuracy of rating prediction in terms of RMSE or the quality of top-$k$ recommendation lists in terms of precision, recall, MAP, etc. However, an important expectation for commercial recommendation systems is to improve the final revenue/profit of the system. Traditional recommendation targets such as rating prediction and top-$k$ recommendation are not directly related to this goal. In this work, we blend the fundamental concepts in online advertising and micro-economics into person…
Deep Neural Networks (DNNs) are susceptible to model stealing attacks, which allows a data-limited adversary with no knowledge of the training dataset to clone the functionality of a target model, just by using black-box query access. Such attacks are typically carried out by querying the target model using inputs that are synthetically generated or sampled from a surrogate dataset to construct a labeled dataset. The adversary can use this labeled dataset to train a clone model, which achieves a classification accuracy comparable to that of the target model. We propose "Adaptive Misinformation…
Georgia Institute of Technology（佐治亚理工学院）；University of Michigan（密歇根大学）
Model Stealing (MS) attacks allow an adversary with black-box access to a Machine Learning model to replicate its functionality, compromising the confidentiality of the model. Such attacks train a clone model by using the predictions of the target model for different inputs. The effectiveness of such attacks relies heavily on the availability of data necessary to query the target model. Existing attacks either assume partial access to the dataset of the target model or availability of an alternate dataset with semantic similarities. This paper proposes MAZE -- a data-free model stealing attack…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Saarland Informatics Campus, Germany（德国萨尔州信息学园区）
High-performance Deep Neural Networks (DNNs) are increasingly deployed in many real-world applications e.g., cloud prediction APIs. Recent advances in model functionality stealing attacks via black-box access (i.e., inputs in, predictions out) threaten the business model of such applications, which require a lot of time, money, and effort to develop. Existing defenses take a passive role against stealing attacks, such as by truncating predicted information. We find such passive defenses ineffective against DNN stealing attacks. In this paper, we propose the first defense which actively perturb…
Rafael - Advanced Defense Systems Ltd.（拉斐尔先进防御系统有限公司）；Department of Electrical Engineering, Technion（以色列理工学院电气工程系）
This paper makes a substantial step towards cloning the functionality of black-box models by introducing a Machine learning (ML) architecture named Deep Neural Trees (DNTs). This new architecture can learn to separate different tasks of the black-box model, and clone its task-specific behavior. We propose to train the DNT using an active learning algorithm to obtain faster and more sample-efficient training. In contrast to prior work, we study a complex "victim" black-box model based solely on input-output interactions, while at the same time the attacker and the victim model may have complete…
Efficient Model-Stealing Attacks Against Inductive Graph Neural Networks
IDEAS NCBR；Institute of Fundamental Technological Research, Polish Academy of Sciences（波兰科学院基础技术研究所）；Warsaw University of Technology（华沙理工大学）；CISPA（CISPA亥姆霍兹信息安全中心）；University of Warsaw（华沙大学）
Norton Research Group（诺顿研究组）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；INRIA（法国国家信息与自动化研究所）
Many real-world data come in the form of graphs. Graph neural networks (GNNs), a new family of machine learning (ML) models, have been proposed to fully leverage graph data to build powerful applications. In particular, the inductive GNNs, which can generalize to unseen data, become mainstream in this direction. Machine learning models have shown great potential in various tasks and have been deployed in many real-world scenarios. To train a good model, a large amount of data as well as computational resources are needed, leading to valuable intellectual property. Previous research has shown t…
Machine learning (ML) models may be deemed confidential due to their sensitive training data, commercial value, or use in security applications. Increasingly often, confidential ML models are being deployed with publicly accessible query interfaces. ML-as-a-service ("predictive analytics") systems are an example: Some allow users to train models on potentially sensitive data and charge others for access on a pay-per-query basis. The tension between model confidentiality and public access motivates our investigation of model extraction attacks. In such attacks, an adversary with black-box acces…
Training deep neural networks often forces users to work in a distributed or outsourced setting, accompanied with privacy concerns. Split learning aims to address this concern by distributing the model among a client and a server. The scheme supposedly provides privacy, since the server cannot see the clients' models and inputs. We show that this is not true via two novel attacks. (1) We show that an honest-but-curious split learning server, equipped only with the knowledge of the client neural network architecture, can recover the input samples and obtain a functionally similar model to the c…
Unifund CCR, LLC（Unifund CCR有限责任公司）；Pamplin College of Business, Virginia Polytechnic Institute（弗吉尼亚理工学院庞普林商学院）；College of Business, University of Cincinnati（辛辛那提大学商学院）
This paper proposes a two-stage scoring approach to help lenders decide their fund allocations in the peer-to-peer (P2P) lending market. The existing scoring approaches focus on only either probability of default (PD) prediction, known as credit scoring, or profitability prediction, known as profit scoring, to identify the best loans for investment. Credit scoring fails to deliver the main need of lenders on how much profit they may obtain through their investment. On the other hand, profit scoring can satisfy that need by predicting the investment profitability. However, profit scoring comple…
Qualcomm AI Research（高通人工智能研究院）；Eindhoven University of Technology（埃因霍温理工大学）
Neural data compression has been shown to outperform classical methods in terms of $RD$ performance, with results still improving rapidly. At a high level, neural compression is based on an autoencoder that tries to reconstruct the input instance from a (quantized) latent representation, coupled with a prior that is used to losslessly compress these latents. Due to limitations on model capacity and imperfect optimization and generalization, such models will suboptimally compress test data in general. However, one of the great strengths of learned compression is that if the test-time data distr…
Pennsylvania State University（宾夕法尼亚州立大学）；Zhejiang University（浙江大学）
Recent years have witnessed the emergence of a new paradigm of building natural language processing (NLP) systems: general-purpose, pre-trained language models (LMs) are composed with simple downstream models and fine-tuned for a variety of NLP tasks. This paradigm shift significantly simplifies the system development cycles. However, as many LMs are provided by untrusted third parties, their lack of standardization or regulation entails profound security implications, which are largely unexplored. To bridge this gap, this work studies the security threats posed by malicious LMs to NLP systems…
