Selective labels are a common feature of consequential decision-making applications, referring to the lack of observed outcomes under one of the possible decisions. This paper reports work in progress on learning decision policies in the face of selective labels. The setting considered is both a simplified homogeneous one, disregarding individuals' features to facilitate determination of optimal policies, and an online one, to balance costs incurred in learning with future utility. For maximizing discounted total reward, the optimal policy is shown to be a threshold policy, and the problem is one of optimal stopping. In contrast, for undiscounted infinite-horizon average reward, optimal policies have positive acceptance probability in all states. Future work stemming from these results is discussed.
Although deep Reinforcement Learning (RL) has proven successful in a wide range of tasks, one challenge it faces is interpretability when applied to real-world problems. Saliency maps are frequently used to provide interpretability for deep neural networks. However, in the RL domain, existing saliency map approaches are either computationally expensive and thus cannot satisfy the real-time requirement of real-world scenarios or cannot produce interpretable saliency maps for RL policies. In this work, we propose an approach of Distillation with selective Input Gradient Regularization (DIGR) whi…
Self-Composing Policies for Scalable Continual Reinforcement Learning
Department of Computer Science and Artificial Intelligence, University of the Basque Country UPV/EHU, Donostia-San Sebastian, Spain（计算机科学与人工智能系，巴斯克国家大学UPV/EHU，圣地亚哥-桑塞班西斯，西班牙）；Basque Center for Applied Mathematics (BCAM), Bilbao, Spain（应用数学巴斯克中心（BCAM），毕尔巴鄂，西班牙）
This work studies the statistical limits of uniform convergence for offline policy evaluation (OPE) problems with model-based methods (for episodic MDP) and provides a unified framework towards optimal learning for several well-motivated offline tasks. Uniform OPE $\sup_Π|Q^π-\hat{Q}^π|<ε$ is a stronger measure than the point-wise OPE and ensures offline learning when $Π$ contains all policies (the global class). In this paper, we establish an $Ω(H^2 S/d_mε^2)$ lower bound (over model-based family) for the global uniform OPE and our main result establishes an upper bound of $\tilde{O}(H^2/d_mε…
The current work on reinforcement learning (RL) from demonstrations often assumes the demonstrations are samples from an optimal policy, an unrealistic assumption in practice. When demonstrations are generated by sub-optimal policies or have sparse state-action pairs, policy learned from sub-optimal demonstrations may mislead an agent with incorrect or non-local action decisions. We propose a new method called Local Ensemble and Reparameterization with Split and Merge of expert policies (LEARN-SAM) to improve efficiency and make better use of the sub-optimal demonstrations. First, LEARN-SAM em…
Conditional Computation in Neural Networks for faster models
School of Computer Science（计算机学院）；McGill University（麦吉尔大学）
深度学习已成为许多应用中最先进的工具，但深度模型的评估和训练可能非常耗时且计算成本高昂。已经提出了条件计算方法来解决这个问题（Bengio等人，2013; Davis & Arel，2013）。它通过一次选择性地只激活网络的一部分来运行。在本文中，我们使用强化学习作为优化条件计算策略的工具。更具体地说，我们将学习依赖于激活的策略以丢弃单元块的问题作为强化学习问题。我们提出了一个由计算速度驱动的学习方案，捕获了想要在保持预测准确性的同时具有简约激活的想法。我们应用一个策略梯度算法来学习优化这个损失函数的策略，并提出了一个正则化机制，鼓励辍学政策的多样化。我们提出了令人鼓舞的实证结果表明，这种方法提高了计算速度，而不影响近似的质量。
Confidence-aware Contrastive Learning for Selective Classification
National Key Laboratory for Novel Software Technology（计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；College of Computer Science and Software Engineering, Hohai University（河海大学计算机与软件学院）；Key Laboratory of Water Big Data Technology of Ministry of Water Resources（水利部水大数据技术重点实验室）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Indian Institute of Technology Madras（马德拉斯印度理工学院）
Robust Policy Search is the problem of learning policies that do not degrade in performance when subject to unseen environment model parameters. It is particularly relevant for transferring policies learned in a simulation environment to the real world. Several existing approaches involve sampling large batches of trajectories which reflect the differences in various possible environments, and then selecting some subset of these to learn robust policies, such as the ones that result in the worst performance. We propose an active learning based framework, EffAcTS, to selectively choose model pa…
MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；Saarland University（萨尔大学）；MPI for Software Systems（马克斯·普朗克软件系统研究所）；ETH Zürich（苏黎世联邦理工学院）
Decision making algorithms, in practice, are often trained on data that exhibits a variety of biases. Decision-makers often aim to take decisions based on some ground-truth target that is assumed or expected to be unbiased, i.e., equally distributed across socially salient groups. In many practical settings, the ground-truth cannot be directly observed, and instead, we have to rely on a biased proxy measure of the ground-truth, i.e., biased labels, in the data. In addition, data is often selectively labeled, i.e., even the biased labels are only observed for a small fraction of the data that r…
