占用信息比率：无限时域、信息导向的参数化策略搜索
Occupancy Information Ratio: Infinite-Horizon, Information-Directed, Parameterized Policy Search
AI总结：
本文提出占用信息比率（OIR）作为无限时域强化学习的新目标，利用拟凹优化与线性规划理论将其转化为凹规划问题，并推导无模型策略梯度算法，在稀疏奖励环境下优于传统方法。
摘要：
在本工作中，我们提出了一种用于无限时域强化学习（RL）的信息导向目标，称为占用信息比率（OIR），其灵感来源于先前用于多臂老虎机和马尔可夫决策过程的信息导向采样方案中的信息比率目标，以及近期通用效用强化学习的进展。OIR由策略的平均成本与其诱导的状态占用测度的熵之间的比率组成，具有丰富的底层结构，并提供了一个可自然应用可扩展、无模型策略搜索方法的目标。具体而言，我们通过利用拟凹优化与马尔可夫决策过程的线性规划理论之间的联系，证明了当底层模型已知时，OIR问题可以转化为凹规划方法并求解。由于在实践中通常缺乏模型知识，我们通过建立相应的策略梯度定理，为无模型OIR策略搜索方法奠定了基础。基于这一结果，我们随后推导了REINFORCE风格和演员-评论家风格的算法，用于在策略参数空间中求解OIR问题。关键在于，利用OIR问题凹规划转化所隐含的强隐藏拟凹性，我们在适当条件下建立了REINFORCE风格方案有限时间收敛到全局最优，以及演员-评论家风格方案渐近收敛到（接近）全局最优。最后，我们通过实验展示了在稀疏奖励设置中，基于OIR的方法相对于普通方法的实用性，支持OIR作为现有RL目标的替代方案。
英文摘要：
In this work, we propose an information-directed objective for infinite-horizon reinforcement learning (RL), called the occupancy information ratio (OIR), inspired by the information ratio objectives used in previous information-directed sampling schemes for multi-armed bandits and Markov decision processes as well as recent advances in general utility RL. The OIR, comprised of a ratio between the average cost of a policy and the entropy of its induced state occupancy measure, enjoys rich underlying structure and presents an objective to which scalable, model-free policy search methods naturally apply. Specifically, we show by leveraging connections between quasiconcave optimization and the linear programming theory for Markov decision processes that the OIR problem can be transformed and solved via concave programming methods when the underlying model is known. Since model knowledge is typically lacking in practice, we lay the foundations for model-free OIR policy search methods by establishing a corresponding policy gradient theorem. Building on this result, we subsequently derive REINFORCE- and actor-critic-style algorithms for solving the OIR problem in policy parameter space. Crucially, exploiting the powerful hidden quasiconcavity property implied by the concave programming transformation of the OIR problem, we establish finite-time convergence of the REINFORCE-style scheme to global optimality and asymptotic convergence of the actor-critic-style scheme to (near) global optimality under suitable conditions. Finally, we experimentally illustrate the utility of OIR-based methods over vanilla methods in sparse-reward settings, supporting the OIR as an alternative to existing RL objectives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
