In this paper, we study the problem of deceptive reinforcement learning to preserve the privacy of a reward function. Reinforcement learning is the problem of finding a behaviour policy based on rewards received from exploratory behaviour. A key ingredient in reinforcement learning is a reward function, which determines how much reward (negative or positive) is given and when. However, in some situations, we may want to keep a reward function private; that is, to make it difficult for an observer to determine the reward function used. We define the problem of privacy-preserving reinforcement learning, and present two models for solving it. These models are based on dissimulation -- a form of deception that `hides the truth'. We evaluate our models both computationally and via human behavioural experiments. Results show that the resulting policies are indeed deceptive, and that participants can determine the true reward function less reliably than that of an honest agent.
New York University（纽约大学）；University of Strathclyde（思克莱德大学）；University of Hertfordshire（赫特福德大学）；Maastricht University（马斯特里赫特大学）
Deep reinforcement learning has learned to play many games well, but failed on others. To better characterize the modes and reasons of failure of deep reinforcement learners, we test the widely used Asynchronous Actor-Critic (A2C) algorithm on four deceptive games, which are specially designed to provide challenges to game-playing agents. These games are implemented in the General Video Game AI framework, which allows us to compare the behavior of reinforcement learning-based agents with planning agents based on tree search. We find that several of these games reliably deceive deep reinforceme…
This paper investigates deceptive reinforcement learning for privacy preservation in model-free and continuous action space domains. In reinforcement learning, the reward function defines the agent's objective. In adversarial scenarios, an agent may need to both maximise rewards and keep its reward function private from observers. Recent research presented the ambiguity model (AM), which selects actions that are ambiguous over a set of possible reward functions, via pre-trained $Q$-functions. Despite promising results in model-based domains, our investigation shows that AM is ineffective in mo…
The Australian National University（澳大利亚国立大学）；Maastricht University（马斯特里赫特大学）
The Angry Birds AI competition has been held over many years to encourage the development of AI agents that can play Angry Birds game levels better than human players. Many different agents with various approaches have been employed over the competition's lifetime to solve this task. Even though the performance of these agents has increased significantly over the past few years, they still show major drawbacks in playing deceptive levels. This is because most of the current agents try to identify the best next shot rather than planning an effective sequence of shots. In order to encourage adva…
Reinforcement learning agents need a reward signal to learn successful policies. When this signal is sparse or the corresponding gradient is deceptive, such agents need a dedicated mechanism to efficiently explore their search space without relying on the reward. Looking for a large diversity of behaviors or using Motion Planning (MP) algorithms are two options in this context. In this paper, we build on the common roots between these two options to investigate the properties of two diversity search algorithms, the Novelty Search and the Goal Exploration Process algorithms. These algorithms lo…
Intention deception involves computing a strategy which deceives the opponent into a wrong belief about the agent's intention or objective. This paper studies a class of probabilistic planning problems with intention deception and investigates how a defender's limited sensing modality can be exploited by an attacker to achieve its attack objective almost surely (with probability one) while hiding its intention. In particular, we model the attack planning in a stochastic system modeled as a Markov decision process (MDP). The attacker is to reach some target states while avoiding unsafe states i…
Safe reinforcement learning has many variants and it is still an open research problem. Here, we focus on how to use action guidance by means of a non-expert demonstrator to avoid catastrophic events in a domain with sparse, delayed, and deceptive rewards: the recently-proposed multi-agent benchmark of Pommerman. This domain is very challenging for reinforcement learning (RL) --- past work has shown that model-free RL algorithms fail to achieve significant learning. In this paper, we shed light into the reasons behind this failure by exemplifying and analyzing the high rate of catastrophic eve…
Deep reinforcement learning has achieved great successes in recent years, however, one main challenge is the sample inefficiency. In this paper, we focus on how to use action guidance by means of a non-expert demonstrator to improve sample efficiency in a domain with sparse, delayed, and possibly deceptive rewards: the recently-proposed multi-agent benchmark of Pommerman. We propose a new framework where even a non-expert simulated demonstrator, e.g., planning algorithms such as Monte Carlo tree search with a small number rollouts, can be integrated within asynchronous distributed deep reinfor…
Coding-Free and Privacy-Preserving Agentic Framework for Data-Driven Clinical Research
Infmedix, Co., Ltd.（Infmedix公司）；Department of Transdisciplinary Studies, Seoul National University（首尔国立大学跨学科研究系）；Healthcare AI Research Institute, Seoul National University Hospital（首尔国立大学医院医疗人工智能研究所）；Department of Medicine, Seoul National University College of Medicine（首尔国立大学医学院医学系）；Department of Transdisciplinary Medicine, Seoul National University Hospital（首尔国立大学医院跨学科医学系）；Department of Radiology, Massachusetts General Hospital and Harvard Medical School（麻省总医院放射科及哈佛医学院）
Massachusetts Institute of Technology (MIT)（麻省理工学院）
We pose an active perception problem where an autonomous agent actively interacts with a second agent with potentially adversarial behaviors. Given the uncertainty in the intent of the other agent, the objective is to collect further evidence to help discriminate potential threats. The main technical challenges are the partial observability of the agent intent, the adversary modeling, and the corresponding uncertainty modeling. Note that an adversary agent may act to mislead the autonomous agent by using a deceptive strategy that is learned from past experiences. We propose an approach that co…
Motivated by personalized healthcare and other applications involving sensitive data, we study online exploration in reinforcement learning with differential privacy (DP) constraints. Existing work on this problem established that no-regret learning is possible under joint differential privacy (JDP) and local differential privacy (LDP) but did not provide an algorithm with optimal regret. We close this gap for the JDP case by designing an $ε$-JDP algorithm with a regret of $\widetilde{O}(\sqrt{SAH^2T}+S^2AH^3/ε)$ which matches the information-theoretic lower bound of non-private learning for a…
Tsinghua University（清华大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Reinforcement learning (RL) algorithms can be used to provide personalized services, which rely on users' private and sensitive data. To protect the users' privacy, privacy-preserving RL algorithms are in demand. In this paper, we study RL with linear function approximation and local differential privacy (LDP) guarantees. We propose a novel $(\varepsilon, δ)$-LDP algorithm for learning a class of Markov decision processes (MDPs) dubbed linear mixture MDPs, and obtains an $\tilde{\mathcal{O}}( d^{5/4}H^{7/4}T^{3/4}\left(\log(1/δ)\right)^{1/4}\sqrt{1/\varepsilon})$ regret, where $d$ is the dimen…
Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学工程学院、计算机科学与工程系）；Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化院）；Shanghai Academy of AI for Science（上海人工智能科学研究院）
Federated prediction for scalable and privacy-preserved knowledge-based planning in radiotherapy
Department of Radiation Oncology, Columbia University Irving Medical Center（放射肿瘤学系，哥伦比亚大学伊文思医疗中心）；Herbert Irving Comprehensive Cancer Center, Columbia University（赫伯特·伊文思综合癌症中心，哥伦比亚大学）；Data Science Institute, Columbia University（数据科学研究所，哥伦比亚大学）
Hong Kong University of Science and Technology（香港科技大学）；University of Guelph（圭尔夫大学）；University of Toronto（多伦多大学）
Businesses planning for the post-pandemic world are looking for innovative ways to protect the health and welfare of their employees and customers. Wireless technologies can play a key role in assisting contact tracing to quickly halt a local infection outbreak and prevent further spread. In this work, we present a wearable proximity and exposure notification solution based on a smartwatch that also promotes safe physical distancing in business, hospitality, or recreational facilities. Our proximity-based privacy-preserving contact tracing (P$^3$CT) leverages the Bluetooth Low Energy (BLE) tec…
Northeast Electric Power University（东北电力大学）；State Grid Jilin Electric Power Company Limited（国网吉林省电力有限公司）
Load forecasting is very essential in the analysis and grid planning of power systems. For this reason, we first propose a household load forecasting method based on federated deep learning and non-intrusive load monitoring (NILM). For all we know, this is the first research on federated learning (FL) in household load forecasting based on NILM. In this method, the integrated power is decomposed into individual device power by non-intrusive load monitoring, and the power of individual appliances is predicted separately using a federated deep learning model. Finally, the predicted power values …
