离线强化学习：状态聚合与轨迹数据的作用
Offline Reinforcement Learning: Role of State Aggregation and Trajectory Data
AI总结：
本文重新审视了无Bellman完备性的离线强化学习问题，证明在多项式样本复杂度下轨迹数据不优于可容许数据，并揭示了离线策略评估的样本复杂度由聚合马尔可夫转移模型中的集中系数决定。
摘要：
我们重新审视了具有价值函数可实现性但缺乏Bellman完备性的离线强化学习问题。Xie和Jiang（2021）以及Foster等人（2022）的先前工作留下了一个开放性问题：有界的集中系数结合基于轨迹的离线数据是否能产生多项式样本复杂度。在这项工作中，我们对离线策略评估任务给出了否定的回答。除了回答这个问题之外，我们还为仅有价值函数可实现性的离线策略评估提供了一个相当完整的图景。我们的主要发现有三点：1）离线策略评估的样本复杂度由聚合马尔可夫转移模型中的集中系数决定，该模型由函数类和离线数据分布共同决定，而不是由原始MDP中的集中系数决定。这统一并推广了Xie和Jiang（2021）以及Foster等人（2022）的思想；2）即使原始MDP中的集中系数较小且离线数据是可容许的（即数据分布等于某个策略的占用度量），聚合马尔可夫转移模型中的集中系数也可能随时间步长呈指数级增长；3）在价值函数可实现性下，存在一种通用的归约方法，可以将任何具有可容许数据的困难实例转换为具有轨迹数据的困难实例，这意味着轨迹数据相比可容许数据没有额外的好处。这三点共同解决了这个开放性问题，尽管它们各自也可能具有独立的意义。
英文摘要：
We revisit the problem of offline reinforcement learning with value function realizability but without Bellman completeness. Previous work by Xie and Jiang (2021) and Foster et al. (2022) left open the question whether a bounded concentrability coefficient along with trajectory-based offline data admits a polynomial sample complexity. In this work, we provide a negative answer to this question for the task of offline policy evaluation. In addition to addressing this question, we provide a rather complete picture for offline policy evaluation with only value function realizability. Our primary findings are threefold: 1) The sample complexity of offline policy evaluation is governed by the concentrability coefficient in an aggregated Markov Transition Model jointly determined by the function class and the offline data distribution, rather than that in the original MDP. This unifies and generalizes the ideas of Xie and Jiang (2021) and Foster et al. (2022), 2) The concentrability coefficient in the aggregated Markov Transition Model may grow exponentially with the horizon length, even when the concentrability coefficient in the original MDP is small and the offline data is admissible (i.e., the data distribution equals the occupancy measure of some policy), 3) Under value function realizability, there is a generic reduction that can convert any hard instance with admissible data to a hard instance with trajectory data, implying that trajectory data offers no extra benefits over admissible data. These three pieces jointly resolve the open problem, though each of them could be of independent interest.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
