Deep reinforcement learning is successful in decision making for sophisticated games, such as Atari, Go, etc. However, real-world decision making often requires reasoning with partial information extracted from complex visual observations. This paper presents Discriminative Particle Filter Reinforcement Learning (DPFRL), a new reinforcement learning framework for complex partial observations. DPFRL encodes a differentiable particle filter in the neural network policy for explicit reasoning with partial observations over time. The particle filter maintains a belief using learned discriminative update, which is trained end-to-end for decision making. We show that using the discriminative update instead of standard generative models results in significantly improved performance, especially for tasks with complex visual observations, because they circumvent the difficulty of modeling complex observations that are irrelevant to decision making. In addition, to extract features from the particle belief, we propose a new type of belief feature based on the moment generating function. DPFRL outperforms state-of-the-art POMDP RL models in Flickering Atari Games, an existing POMDP RL benchmark, and in Natural Flickering Atari Games, a new, more challenging POMDP RL benchmark introduced in this paper. Further, DPFRL performs well for visual navigation with real-world data in the Habitat environment.
STRIDE: Strategic Trajectory Reasoning via Discriminative Estimation for Verifiable Reinforcement Learning
Kean University（基恩大学）；Case Western Reserve University（凯斯西储大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；The Ohio State University（俄亥俄州立大学）；Tongji University（同济大学）；Duke Kunshan University（昆山杜克大学）；Royal Melbourne Institute of Technology（皇家墨尔本理工大学）
One of the classical problems in machine learning and data mining is feature selection. A feature selection algorithm is expected to be quick, and at the same time it should show high performance. MeLiF algorithm effectively solves this problem using ensembles of ranking filters. This article describes two different ways to improve MeLiF algorithm performance with parallelization. Experiments show that proposed schemes significantly improves algorithm performance and increase feature selection quality.
There is a growing interest in using Kalman-filter models in brain modelling. In turn, it is of considerable importance to make Kalman-filters amenable for reinforcement learning. In the usual formulation of optimal control it is computed off-line by solving a backward recursion. In this technical note we show that slight modification of the linear-quadratic-Gaussian Kalman-filter model allows the on-line estimation of optimal control and makes the bridge to reinforcement learning. Moreover, the learning rule for value estimation assumes a Hebbian form weighted by the error of the value estima…
Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptually-specified goals using only a stream of observations and actions. Our agent simultaneously learns a goal-conditioned policy and a goal achievement reward function that measures how similar a state is to the goal state. This dual optimization leads to a co-operative game, giving rise to a learned reward function that reflects similarity in controllable asp…
Automatic machine learning (AutoML) is an area of research aimed at automating machine learning (ML) activities that currently require human experts. One of the most challenging tasks in this field is the automatic generation of end-to-end ML pipelines: combining multiple types of ML algorithms into a single architecture used for end-to-end analysis of previously-unseen data. This task has two challenging aspects: the first is the need to explore a large search space of algorithms and pipeline architectures. The second challenge is the computational cost of training and evaluating multiple pip…
A variety of representation learning approaches have been investigated for reinforcement learning; much less attention, however, has been given to investigating the utility of sparse coding. Outside of reinforcement learning, sparse coding representations have been widely used, with non-convex objectives that result in discriminative representations. In this work, we develop a supervised sparse coding objective for policy evaluation. Despite the non-convexity of this objective, we prove that all local minima are global minima, making the approach amenable to simple optimization strategies. We …
Eindhoven University of Technology（埃因霍温理工大学）；University of Alberta（阿尔伯塔大学）；University of Twente（特文特大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））；University of Luxembourg（卢森堡大学）
Tomorrow's robots will need to distinguish useful information from noise when performing different tasks. A household robot for instance may continuously receive a plethora of information about the home, but needs to focus on just a small subset to successfully execute its current chore. Filtering distracting inputs that contain irrelevant data has received little attention in the reinforcement learning literature. To start resolving this, we formulate a problem setting in reinforcement learning called the $\textit{extremely noisy environment}$ (ENE), where up to $99\%$ of the input features a…
A major challenge in recommender systems is handling new users, whom are also called $\textit{cold-start}$ users. In this paper, we propose a novel approach for learning an optimal series of questions with which to interview cold-start users for movie recommender systems. We propose learning interview questions using Deep Q Networks to create user profiles to make better recommendations to cold-start users. While our proposed system is trained using a movie recommender system, our Deep Q Network model should generalize across various types of recommender systems.
Deep reinforcement learning algorithms have been shown to learn complex tasks using highly general policy classes. However, sparse reward problems remain a significant challenge. Exploration methods based on novelty detection have been particularly successful in such settings but typically require generative or predictive models of the observations, which can be difficult to train when the observations are very high-dimensional and complex, as in the case of raw images. We propose a novelty detection algorithm for exploration that is based entirely on discriminatively trained exemplar models, …
In this paper, we study a multi-step interactive recommendation problem, where the item recommended at current step may affect the quality of future recommendations. To address the problem, we develop a novel and effective approach, named CFRL, which seamlessly integrates the ideas of both collaborative filtering (CF) and reinforcement learning (RL). More specifically, we first model the recommender-user interactive recommendation problem as an agent-environment RL task, which is mathematically described by a Markov decision process (MDP). Further, to achieve collaborative recommendations for …
Technical University of Munich（慕尼黑工业大学）；ETH Zurich（苏黎世联邦理工学院）
Ensuring safety is a crucial challenge when deploying reinforcement learning (RL) to real-world systems. We develop confidence-based safety filters, a control-theoretic approach for certifying state safety constraints for nominal policies learned via standard RL techniques, based on probabilistic dynamics models. Our approach is based on a reformulation of state constraints in terms of cost functions, reducing safety verification to a standard RL task. By exploiting the concept of hallucinating inputs, we extend this formulation to determine a "backup" policy that is safe for the unknown syste…
Beijing National Laboratory for Condensed Matter Physics（北京凝聚态物理国家实验室）；School of Physical Sciences, University of Chinese Academy of Sciences, Beijing 100190, China（中国科学院大学物理科学学院）；Songshan Lake Materials Laboratory, Dongguan, Guangdong 523808, China（松山湖材料实验室）
The application of reinforcement learning algorithms onto real life problems always bears the challenge of filtering the environmental state out of raw sensor readings. While most approaches use heuristics, biology suggests that there must exist an unsupervised method to construct such filters automatically. Besides the extraction of environmental states, the filters have to represent them in a fashion that support modern reinforcement algorithms. Many popular algorithms use a linear architecture, so one should aim at filters that have good approximation properties in combination with linear f…
Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability
Bosch Corporate Research（博世企业研究院）；Technical University Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science（认知科学中心）
EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
Zhongguancun Academy（中关村学院）；College of Software, Nankai University（南开大学软件学院）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
Deep reinforcement learning (DRL) has achieved significant success in various robot tasks: manipulation, navigation, etc. However, complex visual observations in natural environments remains a major challenge. This paper presents Contrastive Variational Reinforcement Learning (CVRL), a model-based method that tackles complex visual observations in DRL. CVRL learns a contrastive variational model by maximizing the mutual information between latent states and observations discriminatively, through contrastive learning. It avoids modeling the complex observation space unnecessarily, as the common…
Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration
School of Electrical；Archimedes, Athena Research Center, Greece；Department of Mathematics, ETH Zurich；Dept. of Mechanical Engineering \& Materials Science, Duke University；Dept. of Digital Systems, University of Piraeus, Greece
Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning
Northwestern Polytechnical University（西北工业大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shenzhen Research Institute of Northwestern Polytechnical University（西北工业大学深圳研究院）；Fudan University（复旦大学）；Harbin Institute of Technology（哈尔滨工业大学）；Tsinghua University（清华大学）；The Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））
Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents
University of Toronto（多伦多大学）；University of Toronto Robotics Institute（多伦多大学机器人研究所）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Technical University of Munich（慕尼黑工业大学）；Munich Institute for Robotics and Machine Intelligence（慕尼黑机器人与机器智能研究所）
We propose discriminative reward co-training (DIRECT) as an extension to deep reinforcement learning algorithms. Building upon the concept of self-imitation learning (SIL), we introduce an imitation buffer to store beneficial trajectories generated by the policy determined by their return. A discriminator network is trained concurrently to the policy to distinguish between trajectories generated by the current policy and beneficial trajectories generated by previous policies. The discriminator's verdict is used to construct a reward signal for optimizing the policy. By interpolating prior expe…
EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
National Engineering Research Center of Software Engineering, Peking University（北京大学软件工程国家工程研究中心）；School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies, Ministry of Education（教育部高可信软件技术重点实验室）；Peking University Information Technology Institute, Tianjin Binhai（北京大学滨海信息技术研究院）；Research Institute, GRGBanking Equipment Co., Ltd.（广电运通金融电子股份有限公司研究院）
University of Stuttgart（斯图加特大学）；Hewlett Packard Enterprise（慧与公司）
Over the last years, supervised learning (SL) has established itself as the state-of-the-art for data-driven turbulence modeling. In the SL paradigm, models are trained based on a dataset, which is typically computed a priori from a high-fidelity solution by applying the respective filter function, which separates the resolved and the unresolved flow scales. For implicitly filtered large eddy simulation (LES), this approach is infeasible, since here, the employed discretization itself acts as an implicit filter function. As a consequence, the exact filter form is generally not known and thus, …
This study addresses the problem of selecting dynamically, at each time instance, the ``optimal'' p-norm to combat outliers in linear adaptive filtering without any knowledge on the potentially time-varying probability distribution function of the outliers. To this end, an online and data-driven framework is designed via kernel-based reinforcement learning (KBRL). Novel Bellman mappings on reproducing kernel Hilbert spaces (RKHSs) are introduced that need no knowledge on transition probabilities of Markov decision processes, and are nonexpansive with respect to the underlying Hilbertian norm. …
Friedrich Alexander Universität Erlangen-Nürnberg（弗里德里希·亚历山大大学埃尔朗根-纽伦堡）
Cone Beam Computed Tomography(CBCT) is a now known method to conduct CT imaging. Especially, The Low Dose CT imaging is one of possible options to protect organs of patients when conducting CT imaging. Therefore Low Dose CT imaging can be an alternative instead of Standard dose CT imaging. However Low Dose CT imaging has a fundamental issue with noises within results compared to Standard Dose CT imaging. Currently, there are lots of attempts to erase the noises. Most of methods with artificial intelligence have many parameters and unexplained layers or a kind of black-box methods. Therefore, o…
University of California, Berkeley, USA（美国加州大学伯克利分校）
Pretraining is a common technique in deep learning for increasing performance and reducing training time, with promising experimental results in deep reinforcement learning (RL). However, pretraining requires a relevant dataset for training. In this work, we evaluate the effectiveness of pretraining for RL tasks, with and without distracting backgrounds, using both large, publicly available datasets with minimal relevance, as well as case-by-case generated datasets labeled via self-supervision. Results suggest filters learned during training on less relevant datasets render pretraining ineffec…
SINTEF Digital（挪威科技工业研究院数字分院）；Norwegian University of Science and Technology（挪威科技大学）
Reinforcement Learning in domains with sparse rewards is a difficult problem, and a large part of the training process is often spent searching the state space in a more or less random fashion for any learning signals. For control problems, we often have some controller readily available which might be suboptimal but nevertheless solves the problem to some degree. This controller can be used to guide the initial exploration phase of the learning controller towards reward yielding states, reducing the time before refinement of a viable policy can be initiated. In our work, the agent is guided t…
In this paper we study how to learn stochastic, multimodal transition dynamics in reinforcement learning (RL) tasks. We focus on evaluating transition function estimation, while we defer planning over this model to future work. Stochasticity is a fundamental property of many task environments. However, discriminative function approximators have difficulty estimating multimodal stochasticity. In contrast, deep generative models do capture complex high-dimensional outcome distributions. First we discuss why, amongst such models, conditional variational inference (VI) is theoretically most appeal…
Universidade Federal de Santa Maria（圣玛丽亚联邦大学）；Universidade Federal de Rio Grande（里奥格兰德联邦大学）；Universidad Tecnológica del Uruguay（乌拉圭技术大学）
Deep Reinforcement Learning (DRL) has produced great achievements since it was proposed, including the possibility of processing raw vision input data. However, training an agent to perform tasks based on image feedback remains a challenge. It requires the processing of large amounts of data from high-dimensional observation spaces, frame by frame, and the agent's actions are computed according to deep neural network policies, end-to-end. Image pre-processing is an effective way of reducing these high dimensional spaces, eliminating unnecessary information present in the scene, supporting the …
Institute of Structural Mechanics（结构力学研究所）；Bauhaus-Universität Weimar（魏玛包豪斯大学）；University of Groningen（格罗宁根大学）
In many reinforcement learning (RL) problems, it takes some time until a taken action by the agent reaches its maximum effect on the environment and consequently the agent receives the reward corresponding to that action by a delay called action-effect delay. Such delays reduce the performance of the learning algorithm and increase the computational costs, as the reinforcement learning agent values the immediate rewards more than the future reward that is more related to the taken action. This paper addresses this issue by introducing an applicable enhanced Q-learning method in which at the be…
University of Oxford（牛津大学）；Microsoft Research（微软研究院）
Sample efficiency is crucial for imitation learning methods to be applicable in real-world applications. Many studies improve sample efficiency by extending adversarial imitation to be off-policy regardless of the fact that these off-policy extensions could either change the original objective or involve complicated optimization. We revisit the foundation of adversarial imitation and propose an off-policy sample efficient approach that requires no adversarial training or min-max optimization. Our formulation capitalizes on two key insights: (1) the similarity between the Bellman equation and t…
Insight Centre for Data Analytics（数据分析洞察中心）；University College Cork（科克大学学院）；University of Edinburgh（爱丁堡大学）；Cankaya University（昌卡亚大学）
Stochastic Constraint Programming (SCP) is an extension of Constraint Programming (CP) used for modelling and solving problems involving constraints and uncertainty. SCP inherits excellent modelling abilities and filtering algorithms from CP, but so far it has not been applied to large problems. Reinforcement Learning (RL) extends Dynamic Programming to large stochastic problems, but is problem-specific and has no generic solvers. We propose a hybrid combining the scalability of RL with the modelling and constraint filtering methods of CP. We implement a prototype in a CP system and demonstrat…
