Off-policy learning, referring to the procedure of policy optimization with access only to logged feedback data, has shown importance in various real-world applications, such as search engines, recommender systems, and etc. While the ground-truth logging policy, which generates the logged data, is usually unknown, previous work simply takes its estimated value in off-policy learning, ignoring both high bias and high variance resulted from such an estimator, especially on samples with small and inaccurately estimated logging probabilities. In this work, we explicitly model the uncertainty in the estimated logging policy and propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning, with a theoretical convergence guarantee. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator against an extensive list of state-of-the-art baselines.
The Chinese University of Hong Kong(香港中文大学);University of Technology Sydney(悉尼科技大学);Australian AI Institute(澳大利亚人工智能研究所);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);Hong Kong Baptist University(香港浸会大学)
Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
University of Florida(佛罗里达大学);UC San Diego(加州大学圣迭戈分校);Northeastern University(东北大学);Northwestern University(西北大学);Stanford University(斯坦福大学);Universität Innsbruck(因斯布鲁克大学)
Probabilistic Margins for Instance Reweighting in Adversarial Training
Hong Kong Baptist University(香港浸会大学);University of Technology Sydney(悉尼科技大学);The University of Sydney(悉尼大学);Nanjing University of Science and Technology(南京理工大学);RIKEN Center for Advanced Intelligence Project(理化学研究所先进智能项目中心);The University of Texas at Austin(德克萨斯大学奥斯汀分校);The University of Tokyo(东京大学)
FaLW: A Forgetting-aware Loss Reweighting for Long-tailed Unlearning
University of Science and Technology of China(USTC)(中国科学技术大学);Suzhou Institute for Advanced Research, USTC(中国科学技术大学苏州研究院);City University of Hong Kong(香港城市大学);Tongji University(同济大学)
RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
Shanghai Jiao Tong University(上海交通大学);Central South University(中南大学);SenseTime Research(商汤科技研究院);The Chinese University of Hong Kong(香港中文大学);Beihang University(北京航空航天大学)
Regret-Aware Policy Optimization: Environment-Level Memory for Replay Suppression under Delayed Harm
Department of Computer Science and Engineering, VTU, Belagavi, India(印度贝尔高姆VTU计算机科学与工程系);Aliens on Earth (AoE) Autonomous Research Group, Belagavi, India(印度贝尔高姆地球外星人自主研究小组)
Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation
Johns Hopkins Carey Business School(约翰斯·霍普金斯凯瑞商学院);International Computer Science Institute(国际计算机科学研究所);Pennsylvania State University(宾夕法尼亚州立大学);Georgia Institute of Technology(佐治亚理工学院)
Improving Generalization and Data Efficiency with Diffusion in Offline Multi-agent RL
Institute for Interdisciplinary Information Sciences(交叉信息学院);Tsinghua University(清华大学);Department of Electronic and Computer Engineering(电子与计算机工程系);Hong Kong University of Science and Technology(香港科学与技术大学)
Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);NLCo Lab, Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院NLCo实验室);Department of Automation, Tsinghua University(清华大学自动化系);Shanghai Jiao Tong University(上海交通大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of California, Los Angeles(加州大学洛杉矶分校)
Promoting Generalization for Exact Solvers via Adversarial Instance Augmentation
CAS Key Laboratory of Technology in GIPAS, University of Science and Technology of China(GIPAS技术CAS重点实验室,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation
Johns Hopkins University(约翰斯·霍普金斯大学);UC Santa Barbara(加州大学圣巴巴拉分校);Deakin University(迪肯大学)
具有线性函数逼近的样本有效离线强化学习(RL)最近被广泛研究。许多先前的工作已经产生了$\tilde{\mathcal{O}}(\frac{1}{\sqrt{K}})$的最小最大最优界限,其中$K$是离线数据中的发作次数。在这项工作中,我们试图了解离线RL与函数近似的实例相关的界限。我们提出了一种名为Bootstrapped and Consolined Pessimistic Value Iteration(BCP-VI)的算法,该算法在悲观主义之上利用了数据自举和约束优化。我们表明,在部分数据覆盖的假设下,\emph{concentrability}相对于最佳的政策,该算法产生了一个快速的$\tilde{\mathcal{O}}(\frac{1}{K})$率离线RL时,有一个积极的差距,在最佳的Q-价值函数,即使当离线数据自适应收集。此外,当最优策略可达状态中的最优动作的线性特征跨越行为策略可达状态并且最优动作是唯一的时,当$K$超过(有限)实例相关阈值时,离线RL实现绝对零次优性误差。据我们所知,这是第一个$\tilde{\mathcal{O}}(\frac{1}{K})$界和绝对零次最优界分别为离线RL与线性函数近似自适应数据的部分覆盖。我们还提供了实例不可知和实例相关的信息理论下限,以补充我们的上限。
Instance-dependent $\ell_\infty$-bounds for policy evaluation in tabular reinforcement learning
Simons Institute for the Theory of Computing, UC Berkeley(加州大学伯克利分校西蒙斯计算理论研究所);Departments of EECS and Statistics, UC Berkeley(加州大学伯克利分校电气工程与计算机科学系及统计学系);Voleon Group, Berkeley(伯克利Voleon集团)
Deep Reinforcement Learning for Traveling Purchaser Problems
Department of Automation \& BNRist, Tsinghua University, Beijing 100084, China;AI Laboratory, Lenovo Research, Beijing 100193, China;School of Computer Science;Engineering, South China University of Technology, Guangzhou 510641, China