轨迹级监督何时允许高效的离线强化学习？
When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?
AI总结：
本文研究离线强化学习中仅使用轨迹级结果（如累积回报或偏好）进行策略优化的统计理论，提出OPAC算法并证明其样本复杂度，同时揭示在非线性聚合目标下存在的统计障碍。
摘要：
离线强化学习通常在过程级奖励监督下进行分析，然而许多序列决策数据集仅记录轨迹级结果。我们发展了从这种结果级监督进行离线策略优化的统计理论。首先研究规范设置，其中目标仍是期望累积奖励，但每个离线轨迹仅提供一个标量标签，其条件均值是累积回报。我们提出OPAC，一种悲观演员-评论家算法，它学习潜在奖励模型并从轨迹级标签优化策略。我们证明了阶为$\widetilde O(H^2\sqrt{C_{sa}(\pi^\star)/n})$的高概率保证和匹配的下界，刻画了用单个轨迹级标签替代过程级奖励的尖锐统计代价。然后我们将该原理扩展到基于偏好的反馈，在偏好模型常数范围内保留了领先的视界和可集中性依赖。最后，我们研究广义基于结果的离线强化学习，其中监督和目标都是由潜在每步奖励的非线性聚合引起的轨迹级量。该问题通常不可学习：对于全成功目标，即使具有确定性转移和常数可集中性，任何离线学习器可能需要$\Omega(2^H)$个轨迹。然后我们通过两个结构系数$\kappa_\mu(\sigma)$和$\chi_\mu(\sigma)$识别出一个可处理的区域，这两个系数捕捉了结果聚合和广义贝尔曼更新中的信息损失，在此区域广义OPAC实现了多项式样本复杂度。我们的结果共同描绘了何时结果级监督能够实现样本高效的离线控制，以及何时缺失过程级奖励会带来根本性的统计障碍。
英文摘要：
Offline reinforcement learning is typically analyzed under process-level reward supervision, yet many sequential decision datasets record only trajectory-level outcomes. We develop a statistical theory for offline policy optimization from such outcome-level supervision. We first study the canonical setting where the target remains the expected cumulative reward, but each offline trajectory provides only a scalar label whose conditional mean is the cumulative return. We propose OPAC, a pessimistic actor-critic algorithm that learns a latent reward model and optimizes a policy from trajectory-level labels. We prove a high-probability guarantee of order $\widetilde O(H^2\sqrt{C_{sa}(π^\star)/n})$ and a matching lower bound, characterizing the sharp statistical cost of replacing process-level rewards with one trajectory-level label. We then extend the principle to preference-based feedback, preserving the leading horizon and concentrability dependence up to preference-model constants. Finally, we study generalized outcome-based offline RL, where both the supervision and the objective are trajectory-level quantities induced by a nonlinear aggregation of latent per-step rewards. This problem is not learnable in general: for all-success objectives, any offline learner may require $Ω(2^H)$ trajectories even with deterministic transitions and constant concentrability. We then identify a tractable regime through two structural coefficients, $κ_μ(σ)$ and $χ_μ(σ)$, capturing information loss in outcome aggregation and generalized Bellman updates, under which generalized OPAC achieves polynomial sample complexity. Together, our results delineate when outcome-level supervision enables sample-efficient offline control and when missing process-level rewards create fundamental statistical barriers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
