Despite recent success of deep network-based Reinforcement Learning (RL), it remains elusive to achieve human-level efficiency in learning novel tasks. While previous efforts attempt to address this challenge using meta-learning strategies, they typically suffer from sampling inefficiency with on-policy RL algorithms or meta-overfitting with off-policy learning. In this work, we propose a novel meta-RL strategy to address those limitations. In particular, we decompose the meta-RL problem into three sub-tasks, task-exploration, task-inference and task-fulfillment, instantiated with two deep network agents and a task encoder. During meta-training, our method learns a task-conditioned actor network for task-fulfillment, an explorer network with a self-supervised reward shaping that encourages task-informative experiences in task-exploration, and a context-aware graph-based task encoder for task inference. We validate our approach with extensive experiments on several public benchmarks and the results show that our algorithm effectively performs exploration for task inference, improves sample efficiency during both training and testing, and mitigates the meta-overfitting problem.
Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether causal reasoning can emerge via meta-reinforcement learning. We train a recurrent network with model-free reinforcement learning to solve a range of problems that each contain causal structure. We find that the trained agent can perform causal reasoning in novel situations in order to obtain rewards. The agent can select informative interventions, draw causal inferences from observational data, and make counterfactual predictions. Although established formal…
Demonstration is an appealing way for humans to provide assistance to reinforcement-learning agents. Most approaches in this area view demonstrations primarily as sources of behavioral bias. But in sparse-reward tasks, humans seem to treat demonstrations more as sources of causal knowledge. This paper proposes a framework for agents that benefit from demonstration in this human-inspired way. In this framework, agents develop causal models through observation, and reason from this knowledge to decompose tasks for effective reinforcement learning. Experimental results show that a basic implement…
University College London（伦敦大学学院）；Delft University of Technology（代尔夫特理工大学）；MediaGamma Ltd.（MediaGamma有限公司）
Humans are capable of attributing latent mental contents such as beliefs or intentions to others. The social skill is critical in daily life for reasoning about the potential consequences of others' behaviors so as to plan ahead. It is known that humans use such reasoning ability recursively by considering what others believe about their own beliefs. In this paper, we start from level-$1$ recursion and introduce a probabilistic recursive reasoning (PR2) framework for multi-agent reinforcement learning. Our hypothesis is that it is beneficial for each agent to account for how the opponents woul…
Cash App Labs（Cash App实验室）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
We convert the DeepMind Mathematics Dataset into a reinforcement learning environment by interpreting it as a program synthesis problem. Each action taken in the environment adds an operator or an input into a discrete compute graph. Graphs which compute correct answers yield positive reward, enabling the optimization of a policy to construct compute graphs conditioned on problem statements. Baseline models are trained using Double DQN on various subsets of problem types, demonstrating the capability to learn to correctly construct graphs despite the challenges of combinatorial explosion and n…
Stanford University（斯坦福大学）；Honda Research Institute US（本田美国研究院）
Multi-agent reinforcement learning (MARL) provides an efficient way for simultaneously learning policies for multiple agents interacting with each other. However, in scenarios requiring complex interactions, existing algorithms can suffer from an inability to accurately anticipate the influence of self-actions on other agents. Incorporating an ability to reason about other agents' potential responses can allow an agent to formulate more effective strategies. This paper adopts a recursive reasoning model in a centralized-training-decentralized-execution framework to help learning agents better …
Reinforcement Learning in hyperbolic space for multi-step reasoning
Texas Tech University Health Science Center（德克萨斯理工大学健康科学中心）；The University of Texas Health Science Center at Houston（德克萨斯大学休斯顿健康科学中心）；Society of Artificial Intelligence Research（人工智能研究学会）
Mitigating Overthinking in Large Reasoning Models via Difficulty-aware Reinforcement Learning
Laboratory for Artificial Intelligence and New Forms of Education（人工智能与新教育形式实验室）；National Engineering Research Center of Educational Big Data（教育大数据国家工程研究中心）；Faculty of Artificial Intelligence in Education, Central China Normal University（教育人工智能学院，中央财经大学）；College of Electronics and Information Engineering, South-Central Minzu University（电子与信息工程学院，西南民族大学）
IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learning
Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Kuaishou Technology（快手科技）；Shenzhen Institute of Advanced Integration Technology, Shenzhen（深圳先进集成技术研究院）
Path-based relational reasoning over knowledge graphs has become increasingly popular due to a variety of downstream applications such as question answering in dialogue systems, fact prediction, and recommender systems. In recent years, reinforcement learning (RL) has provided solutions that are more interpretable and explainable than other deep learning models. However, these solutions still face several challenges, including large action space for the RL agent and accurate representation of entity neighborhood structure. We address these problems by introducing a type-enhanced RL agent that …
Current image-based reinforcement learning (RL) algorithms typically operate on the whole image without performing object-level reasoning. This leads to inefficient goal sampling and ineffective reward functions. In this paper, we improve upon previous visual self-supervised RL by incorporating object-level reasoning and occlusion reasoning. Specifically, we use unknown object segmentation to ignore distractors in the scene for better reward computation and goal generation; we further enable occlusion reasoning by employing a novel auxiliary loss and training scheme. We demonstrate that our pr…
For Artificial Intelligence to have a greater impact in biology and medicine, it is crucial that recommendations are both accurate and transparent. In other domains, a neurosymbolic approach of multi-hop reasoning on knowledge graphs has been shown to produce transparent explanations. However, there is a lack of research applying it to complex biomedical datasets and problems. In this paper, the approach is explored for drug discovery to draw solid conclusions on its applicability. For the first time, we systematically apply it to multiple biomedical datasets and recommendation tasks with fair…
University of Science and Technology of China（中国科学技术大学）；SUNY Binghamton（纽约州立大学宾汉姆顿分校）；UT Austin（德克萨斯大学奥斯汀分校）
Reinforcement learning (RL) agents aim at learning by interacting with an environment, and are not designed for representing or reasoning with declarative knowledge. Knowledge representation and reasoning (KRR) paradigms are strong in declarative KRR tasks, but are ill-equipped to learn from such experiences. In this work, we integrate logical-probabilistic KRR with model-based RL, enabling agents to simultaneously reason with declarative knowledge and learn from interaction experiences. The knowledge from humans and RL is unified and used for dynamically computing task-specific planning model…
Command and Control Engineering College, Army Engineering University of PLA（中国人民解放军陆军工程大学指挥控制工程学院）；troop of PLA（中国人民解放军31436部队）；Academy of Military Science（军事科学院）
In general, multiple domain cyberspace security assessments can be implemented by reasoning user's permissions. However, while existing methods include some information from the physical and social domains, they do not provide a comprehensive representation of cyberspace. Existing reasoning methods are also based on expert-given rules, resulting in inefficiency and a low degree of intelligence. To address this challenge, we create a Knowledge Graph (KG) of multiple domain cyberspace in order to provide a standard semantic description of the multiple domain cyberspace. Following that, we propos…
Stanford University（斯坦福大学）；Toyota Research Institute（丰田研究所）；Honda Research Institute（本田研究所）
Maneuvering in dense traffic is a challenging task for autonomous vehicles because it requires reasoning about the stochastic behaviors of many other participants. In addition, the agent must achieve the maneuver within a limited time and distance. In this work, we propose a combination of reinforcement learning and game theory to learn merging behaviors. We design a training curriculum for a reinforcement learning agent using the concept of level-$k$ behavior. This approach exposes the agent to a broad variety of behaviors during training, which promotes learning policies that are robust to m…
Fishing Out Free Riders: Shapley-Based Reward Attribution for Parallel Reasoning via Reinforcement Learning
ShanghaiTech University（上海科技大学）；City University of Hong Kong（香港城市大学）；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Zhejiang University（浙江大学）
Reinforcement Learning Improves LLM Accuracy and Reasoning in Disease Classification from Radiology Reports
Department of Population Health Sciences, Weill Cornell Medicine, New York, NY（人口健康科学系，韦尔·柯林斯医学中心，纽约，NY）；Department of Radiology, Weill Cornell Medicine, New York, NY（放射科，韦尔·柯林斯医学中心，纽约，NY）；Department of Radiology, Thomas Jefferson University, Philadelphia, PA（放射科，托马斯·杰斐逊大学，费城，PA）
What is the objective of reasoning with reinforcement learning?
Department of Statistics and Data Science, The Wharton School, University of Pennsylvania（统计与数据科学系，沃森商学院，宾夕法尼亚大学）；Department of Electrical Engineering and Computer Sciences, University of California, Berkeley（电气工程与计算机科学系，加州大学伯克利分校）
Recent advances in personalized recommendation have sparked great interest in the exploitation of rich structured information provided by knowledge graphs. Unlike most existing approaches that only focus on leveraging knowledge graphs for more accurate recommendation, we perform explicit reasoning with knowledge for decision making so that the recommendations are generated and supported by an interpretable causal inference procedure. To this end, we propose a method called Policy-Guided Path Reasoning (PGPR), which couples recommendation and interpretability by providing actual paths in a know…
