历史依赖日志记录下离线策略评估的指数级困难性

中文摘要 AI 辅助

一份记录数据集能否频繁访问每个隐藏状态，却仍然对目标策略的价值提供指数级不充分的信息？我们证明，当记录器依赖于历史时，这是可能的。对于每个时间范围 $H \ge 3$，我们构造两个POMDP，每个阶段最多有两个潜在状态，三个动作，以及一个具有三个记忆状态的公共记录器。动作覆盖、信念覆盖以及两个行为边际结果揭示条件都具有与 $H$ 无关的常数。然而，即使两个候选模型都是已知的，评估一个已知的确定性目标策略达到 $1/8$ 的精度，在置信度 $1-\delta$ 下（对于 $0 < \delta \le 1/4$）需要 $\Theta((3/2)^H \log(1/\delta))$ 个记录片段。其机制很简单：一次重置会抹去决定目标值的未知转移。我们精确刻画了由此产生的统计实验，并获得了匹配的最优估计器。一个定向的双车道网格世界实现了该构造，轨迹模拟与其有限样本预测一致。该结果确立了在Zhang和Jiang（2025，arXiv:2503.01134）提出的历史依赖记录、基于模型的情况下，在其行为边际揭示定义下的难解性。

英文摘要

Can a logged dataset visit every hidden state frequently and still be exponentially uninformative about a target policy's value? We show that it can when the logger depends on history. For every horizon $H \ge 3$, we construct two POMDPs with at most two latent states per stage, three actions, and a common logger with three memory states. Action coverage, belief coverage, and two behavior-marginal outcome-revealing conditions all have constants independent of $H$. Nevertheless, evaluating a known deterministic target policy to accuracy $1/8$ requires $Θ((3/2)^H \log(1/δ))$ logged episodes at confidence $1-δ$, for $0 < δ\le 1/4$, even when both candidate models are known. The mechanism is simple: a reset erases the unknown transition that determines the target value. We characterize the resulting statistical experiment exactly and obtain a matching optimal estimator. A directed two-lane gridworld realizes the construction, and trajectory simulations agree with its finite-sample prediction. The result establishes intractability for the history-dependent-logging, model-based case posed by Zhang and Jiang (2025, arXiv:2503.01134), under their behavior-marginal definition of revealing.

发表机构

• University of Alberta（阿尔伯塔大学）

机构由 AI 辅助整理，请以论文原文为准。