Eliška Zugarová, Tatiana V. Guy
A problem of learning decision policy from past experience is considered. Using the Fully Probabilistic Design (FPD) formalism, we propose a new general approach for finding a stochastic policy from the past data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
FAST: Similarity-based Knowledge Transfer for Efficient Policy Learning
本文提出 FAST 框架，利用视觉帧与文本描述估计任务相似性并选择可迁移策略，从而在赛道任务中以更少训练步骤达到接近从零学习的性能。
Learning generalizeable policies from visual input in the presence of visual distractions is a challenging problem in reinforcement learning. Recently, there has been renewed interest in bisimulation metrics as a tool to address this issue; these metrics can be used to learn representations that are, in principle, invariant to irrelevant distractions by measuring behavioural similarity between states. An accurate, unbiased, and scalable estimation of these metrics has proved elusive in continuous state and action scenarios. We propose entangled bisimulation, a bisimulation metric that allows t…
Explainable Representation of Finite-Memory Policies for POMDPs using Decision Trees
本文提出通过决策树和Mealy机结合的方法，为POMDPs中的有限内存策略提供可解释且更小的表示，提升策略的可解释性。
Optimal Decision Tree Policies for Markov Decision Processes
针对强化学习策略可解释性难题，提出基于混合整数线性规划的OMDT方法，直接为MDP优化尺寸受限决策树以最大化期望折扣回报，发现其性能接近最优且优于模仿学习。
Reinforcementlearning(RL)folkloresuggeststhathistory-basedfunctionapproximationmethods,suchas recurrent neural nets or history-based state abstraction, perform better than their memory-less counterparts, due to the fact that function approximation in Markov decision processes (MDP) can be viewed as inducing a Partially observable MDP. However, there has been little formal analysis of such history-based algorithms, as most existing frameworks focus exclusively on memory-less features. In this paper, we introduce a theoretical framework for studying the behaviour of RL algorithms that learn to c…
On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes
我们考虑无限时域平稳$γ$折扣马尔可夫决策过程，它是已知的，存在一个平稳的最优策略。使用在每次迭代时具有一些误差$ε$的值和策略迭代，众所周知，可以计算$\frac{2γ}{(1-γ)^2}ε$最优的固定策略。在认为这种保证是紧的，我们开发的变化值和政策迭代计算非平稳的政策，可以达到$\frac{2γ}{1-γ}ε$ -最优的，这构成了一个显着的改善通常情况下，当$γ$接近1。令人惊讶的是，这表明“计算近最优非平稳策略”的问题比“计算近最优平稳策略”的问题简单得多。
We address the problem of learning a decision policy from observational data of past decisions in contexts with features and associated outcomes. The past policy maybe unknown and in safety-critical applications, such as medical decision support, it is of interest to learn robust policies that reduce the risk of outcomes with high costs. In this paper, we develop a method for learning policies that reduce tails of the cost distribution at a specified level and, moreover, provide a statistically valid bound on the cost of each decision. These properties are valid under finite samples -- even in…
Reinforcement learning methods trained on few environments rarely learn policies that generalize to unseen environments. To improve generalization, we incorporate the inherent sequential structure in reinforcement learning into the representation learning process. This approach is orthogonal to recent approaches, which rarely exploit this structure explicitly. Specifically, we introduce a theoretically motivated policy similarity metric (PSM) for measuring behavioral similarity between states. PSM assigns high similarity to states for which the optimal policies in those states as well as in fu…
Non-Deterministic Policies in Markovian Decision Processes
马尔可夫过程长期以来一直被用来模拟随机环境。强化学习已经成为解决此类环境中的顺序规划和决策问题的框架。近年来，人们尝试应用强化学习的方法来构建马尔可夫环境下的行动选择决策支持系统。虽然强化学习中的传统方法已被证明在有关顺序决策的问题中是有用的，但它们不能以其当前形式应用于决策支持系统，例如医疗领域中的决策支持系统，因为它们建议的政策通常是高度规范的，并且几乎没有给用户输入留下空间。如果没有能力提供灵活的指导方针，这些方法就不太可能得到这类系统用户的欢迎。本文介绍了新的概念，非确定性的政策，让更多的灵活性，在用户的决策过程中，同时限制决策保持接近最优的解决方案。我们提供了两个算法来计算离散域中的非确定性策略。我们研究了这些方法的输出和运行时间上的一组合成和现实世界的问题。在与人类受试者的实验中，我们表明，人类的辅助提示基于非确定性的政策优于人类和计算机只代理在网络导航任务。
Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policies
本文提出在参数空间中对不同MuJoCo任务上单独训练的决策Transformer进行权重平均，无需集中式训练即可形成多任务策略，并验证了预训练初始化、模型容量和Fisher信息加权等选择的重要性。
How to Find and Reuse Policies for Continuous Adaptation in Lifelong Reinforcement Learning
本研究提出AMSC方法，利用Wasserstein任务嵌入估计相似性并组合先前策略，在终身强化学习中实现高效迁移，实验显示更高性能且无遗忘。
In this paper we investigate the use of MPC-inspired neural network policies for sequential decision making. We introduce an extension to the DAgger algorithm for training such policies and show how they have improved training performance and generalization capabilities. We take advantage of this extension to show scalable and efficient training of complex planning policy architectures in continuous state and action spaces. We provide an extensive comparison of neural network policies by considering feed forward policies, recurrent policies, and recurrent policies with planning structure inspi…
Current work in explainable reinforcement learning generally produces policies in the form of a decision tree over the state space. Such policies can be used for formal safety verification, agent behavior prediction, and manual inspection of important features. However, existing approaches fit a decision tree after training or use a custom learning procedure which is not compatible with new learning techniques, such as those which use neural networks. To address this limitation, we propose a novel Markov Decision Process (MDP) type for learning decision tree policies: Iterative Bounding MDPs (…
Combinations and Mixtures of Optimal Policies in Unichain Markov Decision Processes are Optimal
我们表明，在单链马尔可夫决策过程中的最优（平稳）政策的组合是最优的。也就是说，设M是一个具有状态空间S，动作空间A和策略π_j^* 的单链马尔可夫决策过程：S -> A（1 \leq j \leq n），具有最优的平均无限时域报酬。然后这些策略的任何组合π，其中对于S中的每个状态i，存在j使得π（i）=π_j^*（i），也是最优的。此外，我们证明了任何最优策略的混合，其中在状态i的每次访问中，选择最优策略的任意动作π_j^*（i），也会产生最优平均奖励。
Machine Learning community is recently exploring the implications of bias and fairness with respect to the AI applications. The definition of fairness for such applications varies based on their domain of application. The policies governing the use of such machine learning system in a given context are defined by the constitutional laws of nations and regulatory policies enforced by the organizations that are involved in the usage. Fairness related laws and policies are often spread across the large documents like constitution, agreements, and organizational regulations. These legal documents …
Diverse Policies Converge in Reward-free Markov Decision Processe
针对现有多样性强化学习算法缺乏收敛性与效率理论支撑的问题，本文提出统一的研究框架，分析多样化策略训练的收敛性，提出可证明高效的算法，并通过数值实验验证有效性。
SPOT: Scalable Policy Optimization with Trees for Markov Decision Processes
针对马尔可夫决策过程中决策树策略优化难的问题，提出将优化建模为混合整数线性规划、用约简空间分支定界解耦动力学与树约束的SPOT方法，实现了可解释性与可扩展性兼具的高质量策略，速度比现有方法快一个数量级。
Optimizing Interpretable Decision Tree Policies for Reinforcement Learning
针对强化学习中神经网络策略缺乏可解释性的问题，提出决策树策略优化（DTPO）算法，利用策略梯度直接优化完整决策树，实验证明其性能优于模仿学习等基线方法。
Optimized Look-Ahead Tree Policies: A Bridge Between Look-Ahead Tree Policies and Direct Policy Search
提出一种混合策略学习方案，通过直接策略搜索学习节点评分函数来指导小型前瞻树的构建，从而结合直接策略搜索和前瞻树策略的优势。
Learning Personalized Decision Support Policies
本文提出学习个性化决策支持策略的问题，开发Modiste工具，利用随机上下文老虎机技术为每个决策者定制支持方式，实验表明个性化策略优于离线策略，并能以最小性能损失降低成本。
Standard reinforcement learning (RL) aims to find an optimal policy that identifies the best action for each state. However, in healthcare settings, many actions may be near-equivalent with respect to the reward (e.g., survival). We consider an alternative objective -- learning set-valued policies to capture near-equivalent actions that lead to similar cumulative rewards. We propose a model-free algorithm based on temporal difference learning and a near-greedy heuristic for action selection. We analyze the theoretical properties of the proposed algorithm, providing optimality guarantees and de…
While deep reinforcement learning has successfully solved many challenging control tasks, its real-world applicability has been limited by the inability to ensure the safety of learned policies. We propose an approach to verifiable reinforcement learning by training decision tree policies, which can represent complex policies (since they are nonparametric), yet can be efficiently verified using existing techniques (since they are highly structured). The challenge is that decision tree policies are difficult to train. We propose VIPER, an algorithm that combines ideas from model compression and…
Markov decision processes (MDPs) are the defacto frame-work for sequential decision making in the presence ofstochastic uncertainty. A classical optimization criterion forMDPs is to maximize the expected discounted-sum pay-off, which ignores low probability catastrophic events withhighly negative impact on the system. On the other hand,risk-averse policies require the probability of undesirableevents to be below a given threshold, but they do not accountfor optimization of the expected payoff. We consider MDPswith discounted-sum payoff with failure states which repre-sent catastrophic outcomes…
Even though model predictive control (MPC) is currently the main algorithm for insulin control in the artificial pancreas (AP), it usually requires complex online optimizations, which are infeasible for resource-constrained medical devices. MPC also typically relies on state estimation, an error-prone process. In this paper, we introduce a novel approach to AP control that uses Imitation Learning to synthesize neural-network insulin policies from MPC-computed demonstrations. Such policies are computationally efficient and, by instrumenting MPC at training time with full state information, they…
Exploring Critical Testing Scenarios for Decision-Making Policies: An LLM Approach
本文提出一种基于大语言模型的在线测试框架，通过“生成-测试-反馈”流程和多尺度场景生成策略，高效探索决策策略的关键且多样测试场景，并在五个基准上显著优于基线方法。
Optimal Decision Making Under Strategic Behavior
本文研究个体策略性投入努力以影响决策时的最优决策政策问题，通过刻画特征分布变化并利用单调性假设或迭代搜索，提出多项式时间算法以找到高效用的决策政策。
Policies trained via Reinforcement Learning (RL) are often needlessly complex, making them difficult to analyse and interpret. In a run with $n$ time steps, a policy will make $n$ decisions on actions to take; we conjecture that only a small subset of these decisions delivers value over selecting a simple default action. Given a trained policy, we propose a novel black-box method based on statistical fault localisation that ranks the states of the environment according to the importance of decisions made in those states. We argue that among other things, the ranked list of states can help expl…
We consider a new form of reinforcement learning (RL) that is based on opportunities to directly learn the optimal control policy and a general Markov decision process (MDP) framework devised to support these opportunities. Derivations of general classes of our control-based RL methods are presented, together with forms of exploration and exploitation in learning and applying the optimal control policy over time. Our general MDP framework extends the classical Bellman operator and optimality criteria by generalizing the definition and scope of a policy for any given state. We establish the con…
Selective labels are a common feature of consequential decision-making applications, referring to the lack of observed outcomes under one of the possible decisions. This paper reports work in progress on learning decision policies in the face of selective labels. The setting considered is both a simplified homogeneous one, disregarding individuals' features to facilitate determination of optimal policies, and an online one, to balance costs incurred in learning with future utility. For maximizing discounted total reward, the optimal policy is shown to be a threshold policy, and the problem is …
To overcome the curses of dimensionality and modeling of Dynamic Programming (DP) methods to solve Markov Decision Process (MDP) problems, Reinforcement Learning (RL) methods are adopted in practice. Contrary to traditional RL algorithms which do not consider the structural properties of the optimal policy, we propose a structure-aware learning algorithm to exploit the ordered multi-threshold structure of the optimal policy, if any. We prove the asymptotic convergence of the proposed algorithm to the optimal policy. Due to the reduction in the policy space, the proposed algorithm provides rema…
Risk-Aware Decision Policies for Agents Under Noisy Perception
针对感知噪声下智能体决策问题，该研究构建人工生命捕食者-猎物觅食模型，发现不确定性感知策略可提升智能体存活率，揭示了行为随不确定性的状态转变，为含噪声标签的鲁棒学习提供类比。
Black-box AI induction methods such as deep reinforcement learning (DRL) are increasingly being used to find optimal policies for a given control task. Although policies represented using a black-box AI are capable of efficiently executing the underlying control task and achieving optimal closed-loop performance, the developed control rules are often complex and neither interpretable nor explainable. In this paper, we use a recently proposed nonlinear decision-tree (NLDT) approach to find a hierarchical set of control rules in an attempt to maximize the open-loop performance for approximating …
Finding Safe Zones of policies Markov Decision Processes
该研究定义了马尔可夫决策过程策略的安全区概念，证明寻找最优安全区通常计算困难，提出了一种在逃逸概率和安全区规模上均达近2倍近似比、仅需多项式样本复杂度的双准则近似学习算法。
Many recent breakthroughs in multi-agent reinforcement learning (MARL) require the use of deep neural networks, which are challenging for human experts to interpret and understand. On the other hand, existing work on interpretable reinforcement learning (RL) has shown promise in extracting more interpretable decision tree-based policies from neural networks, but only in the single-agent setting. To fill this gap, we propose the first set of algorithms that extract interpretable decision-tree policies from neural networks trained with MARL. The first algorithm, IVIPER, extends VIPER, a recent m…
Distributing entanglement over long distances is one of the central tasks in quantum networks. An important problem, especially for near-term quantum networks, is to develop optimal entanglement distribution protocols that take into account the limitations of current and near-term hardware, such as quantum memories with limited coherence time. We address this problem by initiating the study of quantum network protocols for entanglement distribution using the theory of decision processes, such that optimal protocols (referred to as policies in the context of decision processes) can be found usi…
We present a new behavioural distance over the state space of a Markov decision process, and demonstrate the use of this distance as an effective means of shaping the learnt representations of deep reinforcement learning agents. While existing notions of state similarity are typically difficult to learn at scale due to high computational cost and lack of sample-based algorithms, our newly-proposed distance addresses both of these issues. In addition to providing detailed theoretical analysis, we provide empirical evidence that learning this distance alongside the value function yields structur…
The use of deep neural networks as function approximators has led to striking progress for reinforcement learning algorithms and applications. Yet the knowledge we have on decision boundary geometry and the loss landscape of neural policies is still quite limited. In this paper we propose a framework to investigate the decision boundary and loss landscape similarities across states and across MDPs. We conduct experiments in various games from Arcade Learning Environment, and discover that high sensitivity directions for neural policies are correlated across MDPs. We argue that these high sensi…
Kinematic Tokenization: Optimization-Based Continuous-Time Tokens for Learnable Decision Policies in Noisy Time Series
运动学分词通过优化连续时间表示，提升噪声时间序列中决策策略的学习能力和校准。
Co-Activation Graph Analysis of Safety-Verified and Explainable Deep Reinforcement Learning Policies
本文结合RL策略模型检查与共激活图分析，解释经过安全验证的深度强化学习策略的内部机制，并通过多类实验验证该方法的适用性。
We address the problem of finding an optimal policy in a Markov decision process under a restricted policy class defined by the convex hull of a set of base policies. This problem is of great interest in applications in which a number of reasonably good (or safe) policies are already known and we are only interested in optimizing in their convex hull. We show that this problem is NP-hard to solve exactly as well as to approximate to arbitrary accuracy. However, under a condition that is akin to the occupancy measures of the base policies having large overlap, we show that there exists an effic…
Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
本文提出一种乐观Actor-Critic框架，采用参数化logit匹配回归目标和近似汤普森采样，实现线性MDP中高效的样本复杂度。
From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning
该研究针对离线强化学习中生成模型计算成本高、无法有效利用次优轨迹的问题，提出MeanFlow2Better模型，通过单步流映射优化决策，在D4RL基准上取得优异性能。
Deep Reinforcement Learning (DRL) has recently achieved significant advances in various domains. However, explaining the policy of RL agents still remains an open problem due to several factors, one being the complexity of explaining neural networks decisions. Recently, a group of works have used decision-tree-based models to learn explainable policies. Soft decision trees (SDTs) and discretized differentiable decision trees (DDTs) have been demonstrated to achieve both good performance and share the benefit of having explainable policies. In this work, we further improve the results for tree-…
We consider the challenge of policy simplification and verification in the context of policies learned through reinforcement learning (RL) in continuous environments. In well-behaved settings, RL algorithms have convergence guarantees in the limit. While these guarantees are valuable, they are insufficient for safety-critical applications. Furthermore, they are lost when applying advanced techniques such as deep-RL. To recover guarantees when applying advanced RL algorithms to more complex environments with (i) reachability, (ii) safety-constrained reachability, or (iii) discounted-reward obje…
Reward optimization in fully observable Markov decision processes is equivalent to a linear program over the polytope of state-action frequencies. Taking a similar perspective in the case of partially observable Markov decision processes with memoryless stochastic policies, the problem was recently formulated as the optimization of a linear objective subject to polynomial constraints. Based on this we present an approach for Reward Optimization in State-Action space (ROSA). We test this approach experimentally in maze navigation tasks. We find that ROSA is computationally efficient and can yie…
We study the problem of learning policy of an infinite-horizon, discounted cost, Markov decision process (MDP) with a large number of states. We compute the actions of a policy that is nearly as good as a policy chosen by a suitable oracle from a given mixture policy class characterized by the convex hull of a set of known base policies. To learn the coefficients of the mixture model, we recast the problem as an approximate linear programming (ALP) formulation for MDPs, where the feature vectors correspond to the occupation measures of the base policies defined on the state-action space. We th…
We present a novel sensor-based learning navigation algorithm to compute a collision-free trajectory for a robot in dense and dynamic environments with moving obstacles or targets. Our approach uses deep reinforcement learning-based expert policy that is trained using a sim2real paradigm. In order to increase the reliability and handle the failure cases of the expert policy, we combine with a policy extraction technique to transform the resulting policy into a decision tree format. The resulting decision tree has properties which we use to analyze and modify the policy and improve performance …
Learning Policies for Markov Decision Processes from Data
我们认为学习的政策的马尔可夫决策过程与数据捕获的状态动作对遵循的政策的问题。我们假设该策略属于一类参数化策略，这些策略使用与状态-动作对相关联的特征来定义。这些特征是先验已知的，然而，只有它们中的未知子集可能是相关的。使用最佳拟合所观察的状态-动作样本的$\ell_1$ -正则化逻辑斯谛回归来恢复对应于所观察的目标策略的策略参数。我们建立的估计和原始政策（遗憾）的推广误差和遍历系数的基本马尔可夫链的平均回报之间的差异的界限。为此，我们结合样本复杂性理论和马尔可夫链平稳分布的灵敏度分析。我们的分析表明，要在$O(\sqrtε)$阶内实现后悔，使用$Ω(\log n \cdot poly(1/ε))$阶的训练样本量就足够了，其中$n$是特征的数量。我们证明了我们的方法的有效性合成机器人导航的例子。
Context-Aware Composition of Agent Policies by Markov Decision Process Entity Embeddings and Agent Ensembles
本文提出一种基于模拟的方法，利用知识图谱和实体嵌入表示异构上下文，并通过并行智能体集成按需组合策略，使智能体无需冗长训练即可在不同上下文间无缝切换并成功完成任务。
Formal Verification of Learned Multi-Agent Communication Policies via Decision Tree Distillation
提出通过决策树蒸馏将多智能体强化学习策略转化为可解释模型，并利用PRISM进行形式化验证，确保安全属性转移至原始网络，在无人机编队任务中实现88.9%属性满足率。
Decision making algorithms, in practice, are often trained on data that exhibits a variety of biases. Decision-makers often aim to take decisions based on some ground-truth target that is assumed or expected to be unbiased, i.e., equally distributed across socially salient groups. In many practical settings, the ground-truth cannot be directly observed, and instead, we have to rely on a biased proxy measure of the ground-truth, i.e., biased labels, in the data. In addition, data is often selectively labeled, i.e., even the biased labels are only observed for a small fraction of the data that r…
Predictive Performance Comparison of Decision Policies Under Confounding
本文提出在混杂与不可观测因素下，利用因果推断及离策略评估方法比较决策策略预测性能，并通过可忽略的不确定性区域估计无参数形式假设的遗憾区间。
MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing
该研究提出MetaRoute-Bench框架，构建含180个任务的基准，对比8种路由策略，发现任务感知组合策略成功率优于静态策略，明确路由组合与验证的关键作用，为智能体工作流路由评估提供可复现方法。
Quantifying task-relevant representational similarity using decision variable correlation
本文提出利用决策变量相关性（DVC）量化两种观察者（模型或大脑）在任务相关决策策略上的相似性，发现模型与猴子的相似性较低，且网络性能越高DVC越低。
Learning Decision Policies with Instrumental Variables through Double Machine Learning
针对离线数据中隐藏混杂因素导致的虚假相关性问题，提出DML-IV方法，通过双重机器学习框架减少两阶段工具变量回归偏差，实现高性能策略学习，并具备理论保证与实验优势。
We observe that many system policies that make threshold decisions involving a resource (e.g., time, memory, cores) naturally reveal additional, or implicit feedback. For example, if a system waits X min for an event to occur, then it automatically learns what would have happened if it waited <X min, because time has a cumulative property. This feedback tells us about alternative decisions, and can be used to improve the system policy. However, leveraging implicit feedback is difficult because it tends to be one-sided or incomplete, and may depend on the outcome of the event. As a result, exis…
Hierarchical Decision Making with Structured Policies: A Principled Design via Inverse Optimization
提出一种逆优化框架，将上层目标抽象与下层结构化决策整合，确保下层策略目标与长期任务一致，在资源分配和避障任务中优于端到端RL等方法。
Learning Optimal and Sample-Efficient Decision Policies with Guarantees
本文提出了一种样本高效且具有保证的学习方法，用于在存在隐藏混杂因素的情况下学习最优决策策略，并通过实验验证了其在现实决策中的有效性。
Near-Equivalent Q-learning Policies for Dynamic Treatment Regimes
本文提出一种基于Q学习的框架，通过引入ε容忍度参数，生成多个近等效的治疗策略，识别出治疗无差异区域，适用于单阶段和多阶段决策问题。
Limits of Actor-Critic Algorithms for Decision Tree Policies Learning in IBMDPs
本文研究IBMDP中学习紧凑决策树的深度强化学习方法，指出深度RL在部分可观测设置下即使简单任务也可能失败，并提出将监督分类任务中的最优树学习转化为完全可观测MDP并高效求解的新算法家族。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。