arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.02061cs.LGcs.AIcs.SYeess.SYstat.ML
收藏

具有历史依赖动态上下文的强化学习

Reinforcement Learning with History-Dependent Dynamic Contexts

Google Research(谷歌研究院);CREST, ENSAE(法国国家统计与经济管理学院CREST研究中心)

Guy Tennenholtz, Nadav Merlis, Lior Shani, Martin Mladenov, Craig Boutilier

AI总结:

提出动态上下文马尔可夫决策过程(DCMDPs)框架处理非马尔可夫环境,针对logistic DCMDPs设计基于模型算法并证明遗憾界,在推荐任务中验证有效性。

摘要:

我们引入了动态上下文马尔可夫决策过程(DCMDPs),这是一种用于历史依赖环境的全新强化学习框架,它将上下文MDP框架推广以处理上下文随时间变化的非马尔可夫环境。我们考虑了该模型的特殊情况,重点关注logistic DCMDPs,它通过利用聚合函数来确定上下文转换,从而打破了对历史长度的指数依赖。这种特殊结构使我们能够推导出一种基于置信度上界(upper-confidence-bound)风格的算法,并为其建立了遗憾界。受理论结果启发,我们为logistic DCMDPs引入了一种实用的基于模型的算法,该算法在潜在空间中进行规划,并对历史依赖特征使用乐观策略。我们在推荐任务(使用MovieLens数据)上展示了我们方法的有效性,其中用户行为动态随着推荐而演变。

英文摘要:

We introduce Dynamic Contextual Markov Decision Processes (DCMDPs), a novel reinforcement learning framework for history-dependent environments that generalizes the contextual MDP framework to handle non-Markov environments, where contexts change over time. We consider special cases of the model, with a focus on logistic DCMDPs, which break the exponential dependence on history length by leveraging aggregation functions to determine context transitions. This special structure allows us to derive an upper-confidence-bound style algorithm for which we establish regret bounds. Motivated by our theoretical results, we introduce a practical model-based algorithm for logistic DCMDPs that plans in a latent space and uses optimism over history-dependent features. We demonstrate the efficacy of our approach on a recommendation task (using MovieLens data) where user behavior dynamics evolve in response to recommendations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑